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ARI  Research  Reports  and  Technical  Reports  are  intended  for  sponsors  of 
R&D  tasks  and  for  other  research  and  military  agencies.  Any  findings  ready 
for  implementation  at  the  time  of  publication  are  presented  in  the  last  part 
of  the  Brief.  Upon  completion  of  a  major  phase  of  the  task,  formal  recom¬ 
mendations  for  official  action  normally  are  conveyed  to  appropriate  military 
agencies  by  briefing  or  Disposition  Form. 


FOREWORD 


The  Manpower  and  Educational  Systems  Technical  Area  of  the  Army  Research 
Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  is  concerned  with  im¬ 
proving  individual  and  unit  training  through  research  in  the  design,  method¬ 
ology,  and  implementation  of  instructional  delivery  systems.  One  aspect  of 
this  research  is  to  develop  procedures  for  improving  the  acceptance  and  use 
of  these  training  systems  by  Army  personnel.  This  report  summarizes  a  sys¬ 
tems  model  and  relevant  issues  to  be  used  on  initial  steps  in  understanding 
the  transfer  of  training  technology  from  the  researcher  to  the  Army  user. 

As  a  6.2  effort,  this  research  was  initiated  to  predict  the  outcome  of  the 
transfer  of  game-based  learning  innovations  to  the  operational  environment. 
Work  on  this  effort  was  accomplished  under  Army  Project  2Q162717A764 ,  FY  1979, 
"Evaluation  and  Assessment  of  Training  Technology."  Technical  assistance  was 
received  from  Mr.  Jim  Baker,  Dr.  Leon  Nawrocki,  Dr.  Harry  O'Neil,  Dr.  Halim 
Ozkaptan,  and  Dr.  Joyce  Shields. 
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BRIEF 


Requirement : 

To  improve  individual  and  unit  proficiency  of  Army  personnel  by  devel¬ 
oping  a  systems  model  for  effective  transfer  of  craining  technology  from  re¬ 
searchers  to  Army  users. 


Procedure : 

Technology  transfer  of  Army  training  products  was  defined  through  analy¬ 
ses  of  military  and  nonmilitary  literature  and  data.  The  background  and  his¬ 
tory  leading  up  to  the  current  interest  of  the  Army  in  training  technology 
transfer  were  presented  in  relation  to  pertinent  data  and  critical  events. 

A  systems  model  of  training  technology  transfer  from  both  military  and 
nonmilitary  sources  was  described,  along  with  the  specific  stages  within 
each  activity  of  the  model.  The  presence  or  absence  of  relevant  stages  in 
the  Army  human  resources  research,  development,  test,  and  evaluation  (RDT&E) 
program  relative  to  the  systems  model  was  emphasized.  Discussion  and  sug¬ 
gestions  to  improve  Army  training  technology  transfer  followed. 

Findings : 

Development  of  the  systems  model  revealed  that  several  issues  need  to 
be  addressed  in  Army  training  technology  transfer.  These  issues  are  cate¬ 
gorized  under  general,  specific,  and  recurrent  topics.  Under  the  general 
category,  definitional,  modeling,  and  transfer  issues  need  attention.  Under 
specifics,  the  issues  that  appear  to  need  work  include  an  assessment  of  needs; 
RDT&E  funding;  information  dissemination;  institutionalization;  and  predic¬ 
tion,  monitoring,  evaluation,  and  fe-jdback.  Recurrent  issues  needing  atten¬ 
tion  are  sponsorship  and  a  self-renewal  capability. 

Utilization  of  Findings: 

This  paper  presents  a  conceptual  view  of  the  current  situation  of  Army 
training  technology  transfer.  Application  of  this  systems  model  may  allow 
Army  decisionmakers  to  set  priorities  for  addressing  specific  activities  or 
issues  related  to  the  utilization  of  training  products. 
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OVERVIEW 


Operational  Problem 

Parallel  with  the  demand  for  new,  costly,  and  complex  Army  materiel  sys¬ 
tems  is  the  requirement  for  insuring  parity  in  the  level  of  traininq  neces¬ 
sary  to  operate  and  maintain  these  systems .  This  need  for  improved  Army 
training  has  led  to  the  research,  development,  test,  and  evaluation  (RDT&E) 
of  new  training  products.  However,  informal  comments  and  anecdotal  evidence 
(Appendix  A)  suggest  that  significant  numbers  of  recently  fielded  training 
products  have  not  been  integrated  sufficiently  nor  used  effectively  to  im¬ 
prove  individual  and  unit  proficiency  within  the  Army's  materiel  systems 
(Drucker,  1977;  Shields,  1976).  One  aspect  of  the  problem  concerns  the 
manner  in  which  new  training  technology  is  transferred  from  researchers  and 
developers  to  users.  The  emphasis  on  training  technology  transfer  is  based 
on  the  assumption  that  the  strategies  and  procedures  used  in  introducing  new 
training  products  in  the  field  determine  their  acceptance  and  use. 


Research  Objective 


The  objective  of  a  successful  training  technology  transfer  program  in 
the  Army  is  the  improvement  of  individual  and  unit  proficiency  in  materiel 
systems.  One  means  of  achieving  this  objective  is  the  provision  of  discus¬ 
sion  and  suggestions  for  improving  training  technology  transfer. 


Scope 


The  Department  of  Defense  (DOD)  Technology  Transfer  Consortium  was 
established  in  July  1971  as  a  result  of  congressional  scrutiny  of  the  extent 
of  spinoff  benefits  from  RDT&E,  similar  to  benefits  from  the  NASA  space  pro¬ 
gram  (NASA,  1973;  Tempest  &  Van  Rooy,  1975).  The  consortium’s  purpose  was 
to  transfer  military  R&D  results  to  solutions  of  civilian  problems  (between- 
agency  transfer)  rather  than  to  concentrate  solely  on  military  problems 
(within-agency  transfer) . 

This  paper  focuses  on  the  transfer  of  training  technology  from  the  Army 
human  resources  (HR)  RDT&E  community  to  the  Army  field  user;  consequently, 
the  transfer  of  trainii  technology  from  the  Army  to  agencies  outside  the 
Army  or  DOD  is  not  discussed. ^  Emphasis  is  on  the  role  of  the  U.S.  Army 
Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI)  in  providing 
accepted  and  useful  HR  RDT&E  products  to  the  user,  as  well  as  on  a  systems 


For  discussion  of  "between-  versus  within-agency  transfer,"  see  Jolly  & 
Creighton,  1975;  Millburn,  1979;  Montanarelli,  Jolly,  &  Creighton,  1977; 
Office  of  Technology  Assessment,.  1978;  and  Teich  &  Whartnaby,  1978. 


1 


model  describing  nonsystem,  advanced  RDT&E  with  reference  to  basic  and  ex¬ 
ploratory  development  when  relevant. 


Approach 

To  present  information  about  technology  transfer  from  which  Army  de¬ 
cisionmakers  can  formulate  a  training  technology  transfer  program,  che 
following  format  is  used: 

1.  Background  history  of  critical  events  leading  up  to  the  current 
interest  of  the  Army  in  training  technology  transfer; 

2.  Description  of  a  systems  model  of  training  technology  transfer, 
from  both  military  and  nonmilitary  sources; 

3.  Description  of  specific  stages  within  each  activity  of  the  systems 
model,  emphasizing  the  presence  or  absence  of  relevant  stages  in 
the  current  Army  HR  RDT&E  program  relative  to  the  systems  model; 
and 

4.  Discussion  and  suggestions  to  improve  Army  training  technology 
transfer. 


Definitions 


In  general,  technology  transfer  can  be  defined  as  a  process  by  which 
existing  research  knowledge  is  transferred  operationally  into  useful  proc¬ 
esses,  products,  or  programs  that  fulfill  actual  or  potential  public  or 
private  needs  (Federal  Council  for  Science  and  Technology,  1975) .  vey  words 
in  the  literature  used  to  describe  technology  transfer  are  change  process, 
diffusion  of  innovations,  innovation  dissemination/transfer,  knowledge 
sharing/transfer,  product  utilization,  research  dissemination/utilization, 
and  technology  assessment/sharing/utilization. 

The  user  is  an  individual  and/or  agency  who  (a)  expresses  or  implies 
a  need/requirement  for  a  training  product  or  (b)  must  approve,  or  transmit 
information  of,  the  training  product  prior  to  its  acceptance  and  use  in  the 
field.  The  researcher  is  an  individual  and/or  agency  involved  in  the  RDT&E 
of  a  training  product  in  response  to  an  expressed  or  implied  user  need.  The 
training  product  is  a  research-derived  set  of  procedures  and/or  devices  de¬ 
signed  to  improve  individual  or  unit  proficiency  in  specified  materiel  sys¬ 
tems.  These  three  terms  are  described  more  fully  in  Appendix  B.  For  the 
purposes  of  this  paper,  Army  training  technology  transfer  is  defined  as  a 
process  by  which  the  HR  RDT&E  community  produces  an  accepted  and  useful 
training  product  that  fulfills  the  expressed  or  implicit  requirements  of 
the  Army  user;  the  main  point  is  that  successful  training  technology  trans¬ 
fer  results  in  a  research  product  designed  Lo  produce  a  change.  The  research 
community  may  view  this  change  in  the  way  the  user  applies  the  training 
product;  the  user  may  view  this  change  in  terms  of  produced  results  within 
the  user's  operational  environment  (Table  1). 


Researchers 


Users 


Operational  use  of  product 

Change-mission  performance 

Incorporation  of  product 

Application  of  knowledge 

Adoption  of  results 

Minimum  translation 

Options  for  development  solution 


Improved  effectiveness 
Cost-effective  improvements 
Usable  information 
Benefit  to  decisionmaking 
User  requirements  provided 
Saving  of  money 

Application  of  operational  evidence 


a 

Based  on  a  survey  of  67  Army,  Navy,  Air  Force,  and  contractor  researchers 
and  product  users  (Drucker,  1977). 


Since  the  most  visible  result  of  training  technology  transfer  is  the 
training  product,  interest  has  been  focused  on  the  dissemination  and  eventual 
use  of  the  training  product.  However,  this  report  shows  that  the  training 
technology  transfer  process  involves  both  formulating  the  training  product 
and  introducing  it  to  the  Army  user.  This  system  concept  document,  in  a 
"top-down"  framework,  supports  the  establishment  of  an  Army  training  tech¬ 
nology  transfer  program  and  presents  a  conceptual  view  of  the  current  situa¬ 
tion  of  training  technology  transfer  in  the  Army  and  other  areas.  The  "big 
picture"  of  the  training  technology  transfer  process  will  allow  Army  decision- 
makers  to  determine  which  specific  activities  in  the  program  may  need 
improvement . 


BACKGROUND 

POD  Involvement 

During  the  past  two  decades,  research  and  development  (R&D)  of  major 
materiel  systems  and  associated  training  products  have  become  increasingly 
specialized.  This  specialization  has  brought  a  greater  emphasis  on  the 
interdependence  between  the  R&D  community  and  the  user  community  (Clarkin, 
1978).  For  example,  in  1966,  President  Johnson  stated  that 

Presidents  .  .  .  need  to  show  more  interest  in  what  the  specific 
results  of  research  are  in  their  lifetime,  and  in  their  administra¬ 
tion.  A  great  deal  of  basic  research  has  been  done  .  .  .  but  I 
think  the  time  has  come  to  zero  in  on  the  targets — by  trying  to  get 
our  knowledge  fully  applied.  ...  We  must  make  sure  that  no  life¬ 
saving  discovery  is  locked  up  in  the  laboratory.  (Comroe  &  Dripps, 
1975) 
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This  statement  forecasted  the  Federal  attitude  toward  a  greater  emphasis 
on  the  usefulness  and  applicability  of  research  products. 

The  Department  of  Defense  conducted  a  study,  "Project  Hindsight,"  in 
1966  to  track  the  use  of  specific  military  R&D  products  (Comroe  &  Dripps, 
1975) .  The  study  analyzed  retrospectively  the  development  process  of  20 
weapon  systems  (e.g.,  Polaris  and  Minuteman  missiles,  M102  Howitzer) .  Some 
of  the  study's  conclusions  were  that  (a)  contributions  of  basic,  scientific 
research  were  minimal;  (b)  scientists  contributed  most  effectively  when 
their  effort  was  mission-oriented;  and  (c)  the  time  lag  between  initial 
discovery  and  final  application  was  shortest  when  the  scientist  worked  in 
areas  targeted  by  the  sponsor .  Although  this  study  focused  on  hardware 
products  (e.g.,  materiel,  weapon  sysrems)  rather  than  on  software  products 
(e.g.,  training  programs),  it  indicated  initial  military  involvement  in 
evaluating  the  usefulness  and  relevancy  of  its  research  products.  During 
this  time,  new  terms  were  popularized  in  the  general  R&D  community  to  re¬ 
flect  this  trend:  targeted  research,  mission-oriented  research,  program¬ 
matic  research,  commission-initiated  research,  contract-support  research, 
and  payoff  research.  Congress  and  the  Office  of  Management  and  Budget  re¬ 
acted  accordingly  with  an  increase  in  NIH-supported  contract  research  and 
commission-initiated  research  (Comroe  &  Dripps,  1975).  Thus,  durino  the 
1960's,  the  trend  was  set  for  justifying  applied  research  in  terms  of  user 
needs,  stated  in  terms  of  goals,  targets,  etc. 

In  the  late  1960 's  and  early  1970' s,  Congress  showed  an  increased  in¬ 
terest  in  the  relevancy  of  R&D  products.  For  example,  the  Mansfield  Amend¬ 
ment  to  an  early  1970  appropriations  bill  stated  that  all  defense  R&D  must 
have  a  direct  and  apparent  relationship  to  the  DOD  mission.  This  relation¬ 
ship  apparently  emphasized  the  overall  DOD  mission,  in  the  development  of 
hardware,  weapon  systems  or  subsystems,  or  in  the  maintenance  of  combat 
readiness  through  better  training,  better  personnel  management,  or  better 
manpov.'er  management  (Taylor,  1978).  In  response  to  this  amendment,  each 
of  the  military  services  reviewed  their  R&D  programs  based  on  categories 
which  had  (a)  both  a  direct  and  apparent  relationship,  (b)  an  apparent  re¬ 
lationship,  or  (c)  neither  a  direct  nor  an  apparent  relationship  to  the 
overall  DOD  mission.  This  intraservice  review  resulted  in  a  restructuring 
of  the  military  R&D  program  *rom  the  standpoint  of  both  relevancy  and  em¬ 
phasis  (Taylor,  1978) . 

In  1971,  the  Ginnesburg  Management  Report  recommended  an  increase  in 
R&D  expenditures  in  the  behavioral  and  social  sciences  and  improvement  of 
the  management  in  these  areas.  Some  of  the  findings  of  the  report  were 

(a)  a  lack  of  communication  between  policy  decisionmakers  and  researchers; 

(b)  too  much  control  of  research  by  the  hardware  community;  (c)  too  much 
research  originating  from  the  bottom  up,  rather  than  from  the  top  down; 

(d)  lack  of  involvement  and  control  by  those  responsible  for  the  program; 

(e)  lack  of  responsiveness  to  the  needs  of  the  client;  and  (f)  too  much 
concentration  on  research  in  the  hands  of  particular  groups  (here,  the 
Ginnesburg  report  focused  on  psychologists  and  the  lack  of  involvement  of 
a  wide  variety  of  disciplines  (Rostker,  1978)). 

The  next  major  historical  point  was  the  House  Appropriations  Committee's 
review  and  report  on  the  1976  fiscal  budget  request  (Taylor,  1978) .  The 
committee  recommended  a  50%  reduction  (from  $40  million  to  $20  million) 


4 


within  human  resources  R&D  due  to  the  lack  of  demonstrated  utility  of  re¬ 
search  products  in  this  area.  However,  the  Senate  restored  half  of  the 
$20  million  reduction,  and  the  House  acceded  to  the  Senate's  position  (Tay¬ 
lor,  1978).  Consequently,  the  budget  for  human  resources  R&D  was  kept  in¬ 
tact,  but  Congress  caught  the  attention  of  the  R&D  community.  Since  that 
time,  there  has  been  an  effort  within  the  R&D  management  community  and  within 
certain  segments  of  the  user  community  to  clarify  the  use  of  human  resources 
R&D  (Taylor,  1978). 

In  November  1975,  following  the  final  authorization  of  the  FY  1976  bud¬ 
get,  the  Chairman  of  the  House  Appropriations  Committee  requested  that  the 
General  Accounting  Office  (GAO)  conduct  a  detailed  review  of  the  military 
human  resources  R&D  program  so  that  a  more  thorough  analysis  of  the  FY  1977 
defense  budget  request  could  be  completed  (Reusse,  1976;  Taylor,  1978).  On 
April  22 ,  1977 ,  the  GAO  report  entitled  "Human  Resources  Research  and  De¬ 
velopment  Results  Can  Be  Better  Managed"  was  published. 

The  GAO  conducted  its  evaluation  over  a  period  of  about  18  months.  They 
developed  an  audit  trail  of  human  resources  research,  using  the  technical 
report  as  a  basic  source  to  audit.  They  visited  eight  organizations  within 
DOD  and  identified  374  reports  that  contained  results  which  the  research 
community  deemed  ready  for  immediate  use.  To  complete  the  audit  trail,  the 
GAO  contacted  users  both  by  survey  and  personal  interviews  to  determine  which 
R&D  results  were  used  and  which  were  not  used.  They  found  that  (by  their 
criteria)  56%  of  the  results  had  been  used  and  38%  had  not  been  used,  with 
6%  considered  for  possible  use.  The  GAO  emphasis  was  on  immediate  use  and 
did  not  consider  results  relating  to  the  decisionmaking  process  (Taylor, 

1978) .  Some  of  the  reasons  that  the  GAO  report  found  for  nonuse  were  that 
(a)  users  believed  that  results  were  foi  information  only;  (b)  users  had 
not  seen  the  reports  or  were  unaware  of  the  reported  results;  and  (c)  the 
research  community  questioned  the  results,  or  they  believed  them  to  be  un¬ 
usable  (Taylor,  1978) . 

As  a  result  of  this  study,  GAO  submitted  to  DOD  four  recommendations 
intended  to  improve  the  management  of  research  utilization: 

1.  Develop  criteria  to  identify  R&D  results  with  immediate  (i.e., 
near-term)  use  potential.  The  GAO  used  three  criteria  for  de¬ 
termining  use  potential  of  a  given  research  project,  i.e.,  re¬ 
search  intended  to  support  change  in  education  or  training; 
equipment;  or  regulations,  order,  policy,  doctrine,  or  manuals. 

One  idea  is  to  require  that  each  published  technical  report  in 
the  human  resources  area  provide  a  statement  concerning  the  in¬ 
tended  user  or  user  community  and  the  suggested  use  of  any  product 
or  findings. 

2.  Improve  communication  between  researchers  and  users.  Some  ideas 
are  periodic  utilization  reports;  user  involvement  through  tech¬ 
nology  process;  collocation  of  researchers  with  users;  conferences; 
and  linkage  agents  serving  as  intermediaries  between  R&D  and  user 
communities. 

3.  Establish  a  monitoring  and  feedback  system  for  tracking  utilization. 
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4.  Develop  a  management  mechanism  for  resolution  of  issues  between 
the  researcher  and  the  user  (Reusse,  1976;  Taylor,  1978) . 

Similar  recommendations  were  put  forth  by  the  1976  Defense  Science 
Board  Task  Force  on  Training  Technology  (Alluisi,  1977)  and  the  Laboratory 
Utilization  Report  (White  &  Taylor,  1976),  with  the  additional  recommenda¬ 
tions  of  the  need  for  (a)  cost-benefit  and  performance-effectiveness  analy¬ 
sis  relating  to  research  utilization;  (b)  resolution  of  conflict  of  interest 
between  researcher  and  user;  and  (c)  emphasis  on  the  whole  R&D  process,  from 
requirements  to  utilization  (Taylor,  1978)  . 

A  major  response  by  the  DOD  human  resources  RDT&E  communities  to  the 
growing  Federal  concern  over  the  use  of  research  products  was  the  convening 
of  the  National  Symposium  of  the  Military  Services  on  Utilization  of  People- 
Related  RDT&E  in  June  1977  (Sands  &  Glaser,  1978).  The  purpose  of  the  sym¬ 
posium  wrs  to  assess  the  problems  of  human  resources  RDT&E,  principally 
within  the  military  establishment,  and  to  formulate  recommendations  for  im¬ 
proving  utilization  strategies  of  RDT&E.  Recommendations  from  this  sym¬ 
posium  emphasized  improvement  and/or  increased  activity  in  the  following 
areas:  (a)  use  of  research  products,  (b)  interaction  between  researchers 

and  user  communities,  (c)  accountability  and  responsibility  affecting  the 
use  of  research  products,  (d)  relevancy  to  R&D  requirements,  (e)  implemenra- 
tion  planning,  and  ( f )  reporting  and  dissemination  of  research  findings. 


Army  Involvement 


In  December  1972 ,  the  Military  Requirements  and  Product  Utilization  Of¬ 
fice  was  formed  within  ARI  to  assist  in  cooperative  efforts  between  the  re¬ 
search  technical  areas  and  the  sponsor-users  of  the  results  of  ARI  research. 
The  staff  consisted  of  military  officers  who  performed  liaison  functions. 

The  purposes  of  the  office  were  to  help  define  needs  and  problems  and  to 
promote  the  adoption  and  effective  use  of  the  research  products  (Uhlaner, 
1977).  This  office  was  subsumed  eventually  under  the  Plans,  Programs,  and 
Operations  Office  in  ARI.  A  major  contribution  of  this  office  was  to  heighten 
those  activities  concerned  with  promoting  the  use  of  ARI  research  products, 
e.g. ,  the  Technical  Advisory  Service,  the  Tri-Service  Utilization  Service, 
the  draft  research  utilization  plan,  and  a  task  force  to  study  the  use  of 
research  products. 

Dusek  (1974)  traced  the  history  of  technology  transfer  in  Army  human 
resources  R&D.  Although  Dusek 's  review  emphasized  between-agency  transfer, 
this  historical  survey  is  important  because  it  provides  an  accounting  of 
significant  contributions  developed  by  the  Army  human  resource  RDT&E 
community. 

Two  major  Army  regulations  defining  doctrinal  and  procedural  aspects 
of  personnel  performance  and  training  RDT&E  were  AR  70-1  (1975)  and  AR  70-8 
(1976) .  These  regulations  defined  the  extent  of  ARI  responsibilities  in 
performing  RDT&E,  the  program  formulation  and  approval  process,  the  RDT&E 
categories,  and  the  implementation  of  RDT&E  fundings.  Major  research  funding 
categories  were  delineated  within  ARI's  purview,  namely  basic  (6.1),  explora¬ 
tory  (6.2),  and  nonsystem  advanced  development  (6.3A).  This  last  category 
focuses  primarily  on  non-material-oriented  technology  (e.g.,  for  training 
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products)  which  has  potential  application  to  a  variety  of  systems  rather 
than  to  one  specific  system. 

In  May  1976,  ARI  research  scientists  participated  in  the  Potomac  Chap¬ 
ter  of  the  Human  Factors  Society  annual  symposium  on  training  technology 
transfer,  which  addressed  research  and  policy  problems  in  education  and 
training.  This  meeting  produced  the  explanation  and  tracing  of  training 
from  the  RDT&E  phase  through  the  process  of  translating  the  resultant  tech¬ 
nology  into  implementation  policy  to  the  impact  of  policy  in  training. 

Drucker  (1977)  presented  the  results  of  a  survey  on  military  research 
product  utilization  to  the  MiliJ  ary  Psychology  Division  of  the  American 
Psychological  Association  in  September  1976.  Major  findings  from  this  sur¬ 
vey  were  (a)  that  researchers  and  users  have  different  meanings  for  the 
term  "research  product  utilization,"  (b)  a  breakdown  of  different  descrip¬ 
tions  of  users,  (c)  factors  influencing  the  acceptance  and  use  of  a  research 
product,  and  (d)  discussion  of  evaluation  criteria  for  product  utilization. 
Shields  (1976)  discussed  the  process  of  technology  transfer  in  relation  to 
the  Army's  need  for  improved  (and  timely)  training  products.  Her  review 
presented  deficiencies  in  existing  models  of  technology  transfer  (e.g., 
implementation,  timeliness) ,  factors  to  be  considered  in  an  Army  model  of 
training  technology  transfer  (e.g.,  incentive  system,  organizational  struc¬ 
ture),  and  current  status  and  future  trends  within  the  Army.  Shields'  re¬ 
view  provided  the  framework  from  which  an  Army  model  of  training  technology 
transfer  could  be  developed. 

In  summary,  both  DOD  and  Army  involvement  in  training  technology  trans¬ 
fer  has  focused  on  the  ways  to  insure  the  acceptance  and  usefulness  of  the 
training  research  product.  Although  a  consensus  had  developed  as  >.0  this 
goal,  the  particular  means  or  sequence  of  activities  necessary  and  sufficient 
to  achieve  the  goal  had  not  been  delineated.  Based  on  a  literature  review, 
this  report  describes  important  activities  in  promoting  the  acceptance  and 
usefulness  of  a  training  product.  Deficiency  in  any  one  of  these  activities 
may  result  in  unsuccessful  implementation  of  the  product. 


SYSTEMS  MODEL 

Based  on  a  review  of  the  literature  on  technology  transfer  and  of  Army 
regulations  pertaining  to  HR  RDT&E,  four  major  sequential  activities  were 
considered  to  be  sufficiently  characteristic  of  the  Army  training  technology 
transfer  process  (Figure  1) .  The  activities  are  defined  and  their  associated 
key  words  presented  in  Table  2.  References  are  categorized  by  the  primary 
activity  emphasized  (Appendix  C) .  The  model  was  developed  in  accord  with 
the  following  considerations. 

1.  The  overall  intent  of  the  model  was  primarily  prescriptive,  i.e., 
informing  the  reader  what  is  done,  as  exemplified  in  the  major 
activities  and  their  stages.  Subsequent  discussion  and  suggestions 
were  designed  to  be  predictive ,  i.e.,  informing  the  reader  of  al¬ 
ternative  methodologies  that  may  improve  certain  activities. 

2 .  Primary  focus  was  on  describing  the  appropriate  sequence  of  activi¬ 
ties  involving  the  initial  formulation  of  a  proposed  researchable 
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meetings  with  the  sponsor/user;  and  (b)  use  of  the 
Army  Research  and  Development  Information  System 
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contracted/grant  research,  fundamental  research 


question,  the  translation  and  development  of  this  question  into  a 
prototype  training  product,  and  the  communication  of  the  results  of 
this  product  to  the  user  for  subsequent  implementation.  This  ap¬ 
proach  is  essentially  a  linear  sequence  model  f^r  explanatory  pur¬ 
poses,  although  in  practical  terms  the  flow  of  activities  may  be 
circular  at  any  one  point  within  the  model.  For  example,  a  train¬ 
ing  product  may  be  disseminated  to  tne  sponsor-user  who  initially 
requested  the  RDT&E  of  the  product,  and  the  information  about  the 
training  product  may  be  presented  to  other  interested  military 
agencies  who  may  in  turn  use  this  information  to  formulate  require¬ 
ments  for  RDT&E  of  a  training  product  specific  to  their  needs.  An¬ 
other  example  is  the  monitoring,  evaluating,  and  providing  of  feed¬ 
back  throughout  all  four  activities  of  the  model.  The  researcher (s) 
and/or  user(s)  may  provide  critical  review  of  the  progress  being 
made  in  the  development/fielding  of  the  training  product.  This 
review  can  result  in  a  modification  of  the  activity  based  on  the 
agreements  set  forth  by  both  the  researcher  and  the  sponsor-user. 

3.  The  result  of  each  activity  (research?_ble  question,  training  product, 
and  user  acceptance;  see  Figure  1)  also  represents  the  basis  from 
which  the  subsequent  activity  is  directed.  For  example,  the  research- 
able  question  is  the  basis  for  RDT&E  where  the  result  is  a  training 
product.  Thus,  the  basis  and  results  of  an  activity,  as  presented 
here,  act  as  conceptual  markers  that  indicate  the  point  of  progress 
through  the  Army  training  technology  transfer  model.  These  conceptual 
markers,  in  a  top-down  approach,  can  be  translated  subsequently  into 
official  documentation. 


Analyze  Requirements 

Analysis  of  requirements  involves  the  formulation  of  a  research  plan 
prior  to  development  of  the  training  product.  The  goal  is  a  researchable 
question  that  is  a  well-conceived  alternative  and  that  guides  the  RDT&E  ac¬ 
tivity  in  a  direction  responsive  to  user  needs .  The  literature  review  re¬ 
vealed  four  major  stages  constituting  analysis  of  requirements  (see  Figure  2) . 
Each  of  these  stages  is  explained  more  fully  in  Appendix  D.  These  stages  de¬ 
scribe  the  procedures  conducted  to  formulate  a  RDT&E  effort.  Moreover,  cur¬ 
rent  Army  procedures  are  identified  in  relation  to  each  of  these  stages.  The 
Army  indicators  (official  documentation  and/or  activities)  of  the  analysis 
of  requirements  are  Human  Resource  Needs  (HRN)  statements,  Scientific  Tech¬ 
nical  Objective  Guides  (STOG) ,  DOD-approved  programs,  Military  Themes,  and 
the  Qualitative  Requirements  Information  (QRI)  (see  Army  Regulations  70-8 
and  70-35) . 

One  stage  in  analysis  of  requirements  is  identifying  problems  for  subse¬ 
quent  solutions.  An  example  of  requirements  of  analysis  in  Army  training  was 
conducted  by  the  Board  for  Dynamic  Training  in  1971  by  direction  of  the  Army 
Chief  of  Staff.  The  Board  identified  the  discrepancies  between  what  the  Army 
required  in  its  training  system  and  what  it  was  then  doing  in  the  system.  As 
a  result  of  this  effort,  the  U.S.  Army  Combat  Arms  Training  Board  (CATB)  was 
formed  to  manage  the  implementation  of  significant  new  training  programs. 
Among  these  newly  formed  programs  were  (a)  the  Training  Extension  Course  (TEC) 
program;  (b)  lasers  and  subcaliber  devices;  (c)  career-long,  integrated 
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personnel  management  and  training  system  for  soldiers;  (a)  techniques  in 
training  literature;  and  (e)  a  network  for  exchanging  ideas  and  requirements 
between  operational  units  and  the  training  base  responsible  for  providing 
instructional  materials  and  resources  (Roberts,  Daubek,  &  Johnston,  1977). 

Four  years  later,  a  symposium  was  held  at  Fort  Eustis,  Va. ,  to  identify 
problems  in  instructional  services  provided  by  Army  schools  (Roberts  et  al., 
1977).  The  1975  Instructional  Techno)ogy  Symposium  report  specified  solution 
strategies  that  were  applied  Army-wide  to  create  a  new  agency,  the  Training 
Development  Institute  (TDI)  of  the  U.S.  Army  Training  and  Doctrine  Command 
(TRADOC) .  The  solution  strategies  specified  by  the  1975  report  essentially 
consisted  of  two  programs:  (a)  expert  instructional  technology  consultation 
and  services  to  Army  schools,  and  (b)  comprehensive  staff  and  faculty  train¬ 
ing  programs.  These  programs  have  had  an  impact  on  other  Army  training  de¬ 
velopment  programs,  such  as  self-paced  instruction  and  the  Integrated  Techni¬ 
cal  Documentation  and  Training  (ITDT)  program;  the  latter  is  a  program  to 
change  technical  publications  from  merely  reference  manuals  into  training 
resources  for  soldiers  (see  also  Army  Research,  Development,  and  Acquisition 
Magazine,  1978). 


Research,  Develop,  Test,  and  Evaluate  Solutions 

The  stages  of  the  RDT&E  solutions  activity  are  detailed  in  Army  Regula¬ 
tions  70-1  and  70-8.  Figure  3  presents  a  general  outline  of  the  research 
funding  efforts'  impact  on  non-system- advanced  development  (6.3A).  A  more 
elaborate  breakdown  of  the  training  RDT&E  effort  was  described  by  the  Train¬ 
ing  Development  Working  Group  of  ARI  (1977) .  This  group  applied  the  frame¬ 
work  of  the  Life  Cycle  System  Management  Model  (LCSMM)  (DA  Pamphlet  No.  11-25, 
1975)  of  Army  materiel  system  RDT&E  to  the  training  RDT&E  effort.  As  Fig¬ 
ure  4  shows,  this  model  primarily  emphasizes  what  the  present  paper  subsumes 
under  the  RDT&E  solutions  activity.  The  major  contribution  of  the  LCSMM 
training  model  appears  to  be  as  procedural  markers  or  guidelines  in  conjunc¬ 
tion  with  the  policy  sta cements  in  Army  Regulations  70-1  and  70-8.  The 
major  theme  throughout  the  RDT&E  solution  activity  is  that  a  well-defined 
researcnable  question  should  be  systematically  investigated  and  developed 
into  an  evaluated  training  product,  guided  by  researcher-user  involvement. 


Disseminate  Information 


Dissemination  of  information  about  a  training  product  may  proceed  along 
two  main  paths.  The  first  path  is  one  in  which  communication  is  directed  to 
persons  other  than  the  user  who  initially  requested  the  RDT&E  of  the  training 
product.  The  purposes  here  are  primarily  to  inform  and  secondarily  to  per¬ 
suade.  Such  activities  can  be  found  currently  in  the  Army  Research  and  De¬ 
velopment  Information  System  (ARDIS) ,  which  includes  management  information 
for  DCSRDA  (MIS) ,  and  scientific  and  technical  information  for  scientists 
and  other  interested  individuals  (S&TI) .  Also,  ARI  scientists  publish  tech¬ 
nical  reports,  provide  technical  advisory  service,  and  administer  the  U.S. 
Army  Human  Factors  Research  and  Development  Conference  in  this  regard.  The 
second  path  is  concerned  with  both  informing  and  persuading  the  user  to  ac¬ 
cept  the  training  product.  He^e,  briefings  and  informal  dialogs  are  just  as 
relevant  as  organizational  sponsorship  and  budgetary  considerations.  The 
stages  for  information  dissemination  are  presented  in  Figure  5. 
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A  significant  amount  of  research  has  been  conducted  to  develop  models 
predictive  of  user  acceptance  through  the  process  of  dissemination  of  in¬ 
formation  about  the  training  product.  Thus,  research  has  focused  on  vari¬ 
ables  influential  in  this  process.  As  noted  in  Appendix  B,  the  level  of 
involvement  of  the  user  to  which  communication  is  directed  is  an  important 
variable.  Moreover,  consideration  of  relevant  factors  influencing  this 
process  have  been  cataloged  in  the  literature  (see  references  in  overviews 
and  models  section  for  more  detail) .  For  example.  Table  3  presents  a  repre¬ 
sentative  list  of  relevant  factors  underlying  the  success  cf  the  dissemina¬ 
tion  of  information  for  eventual  user  acceptance  of  the  training  product 
(Shields,  1976) .  These  factors  could  be  applied  as  a  checklist  indicating 
either  the  presence  or  absence  of  these  factors  during  training  technology 
transfer,  or  rated  along  a  dimension  of  visibility,  such  as  completely  ab¬ 
sent,  somewhat  visible,  completely  present. 

Another  use  of  these  factors  has  been  more  quantitative  and  predictive 
in  purpose.  This  approach  attempts  to  translate  qualitative,  descriptive 
factors  deemed  important  for  technology  transfer  into  quantitative,  numeri¬ 
cal  values  which  can  be  incorporated  in  a  mathematical  equation,  whose  re¬ 
sult  would  provide  an  index  representative  of  the  predicted  success/failure 
of  the  acceptance  or  use  of  a  training  product  by  the  user.  An  example  of 
this  approach  is  provided  by  the  Linker  Model  of  Jolly  and  Creighton  (1977) 
presented  in  Table  4.  The  Linker  concept  is  used  in  this  model  as  a  term 
for  effective  technology  transfer.  The  linking  mechanism  necessary  to 
achieve  effective  technology  transfer  is  described  by  identifying  the  fac¬ 
tors  that  contribute  to  predicting  the  movement  of  technology  from  the 
source  of  knowledge  (researcher)  to  the  use  of  knowledge  (user/receiver) . 

The  nine  factors  influential  in  the  dissemination  process  are  divided  into 
two  classes,  formal  and  informal  factors.  Formal  factors  consist  of  pro¬ 
cedures  for  dissemination  of  storr  ,  indexing,  and  retrieval  of  knowledge. 
These  procedures  may  consist  of  different  types  of  informative  reports  de¬ 
signed  for  specific  levels  of  uses,  data  banks,  and/or  user  feedback/input 
documentation  which  influences  the  RDT&E  of  the  training  product.  Formal 
factors  are  generally  visible  and  easy  to  catalog  and  measure.  Informal 
factors  involve  interpersonal  communications  and  contacts;  personal  beliefs 
and  feelings  about  a  knowledge  source;  and  perceptions  about  one's  organiza¬ 
tion,  supervisors,  and  peers.  Informal  factors  are  essentially  behavioral 
in  nature,  measured  by  subjective  reports  that  attempt  to  scale  attitudes 
and  response  evaluations  of  training  technology  transfer.  These  factors  are 
listed  and  described  below. 


Formal  Factors 


1.  Method  of  Information  Documentation:  Refers  to  how  the  technical 
information  is  recorded;  rated  by  the  format  used,  organization  of 
material,  complexity  of  the  language,  ease  of  indexing  and  retrieval 
(e.g.,  ARDIS,  MIS  systems,  executive  summary  style). 

2.  The  Distribution  System:  Refers  to  the  physical  channel  through 
which  information  flows  (e.g.,  formal  distribution  lists,  publica¬ 
tions  in  journals,  symposia  presentations  and  conferences,  informal 
meetings) . 
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Table  3 


Factors  Within  Each  of  the  Major  Categories  of  Variables  Which 
May  Influence  Technology  Transfer  in  the  Army3 


Innovation  Factors 

Type  of  innovation 
(hard  vs.  soft) 
Source  of  innovation 
Match  with  operating 
environment ,  capa¬ 
bilities  of  human 
operators  and  other 
elements  of  the 
system 
Reliability 
Complexity 
Effectiveness 
Documentation 
Ease  of  use 
Cap-tal,  personnel, 
and  time 
requirements 
Visibility 
Adaptability 


Implementation  Factors 


Organizational  Factors 
Size 

Formal  structure 
Informal  structure 
Attitude  toward 
authority 

Goals,  values,  customs 
Cohesive ness 
Communication  patterns 
Communication  with 
change  agent 
Organizational 
inte  rdependency 
Resources 


Individual  Factors 

Education/^raining 
Intellectual  level 
Status  in  group 
Alternatives 
Attitude  toward 
change 

Attitude  toward 
change  agent 
Attitude  toward 
authority 
Self-esteem 
Satisfaction  with 
intrinsic 
motivators 
Cosmopolitanism 
Exposure  to  outside 
attitudes 
Attitude  toward 

present  equipment/ 
system 

Attitude  toward 
innovation 
Resources 


External  Factors — Milieu 


Presence  of  a  change  agent 
Communication  channels  used 
Vigor  of  the  communications  effort 
Form  of  communications  (content) 
Characteristics  of  the  change  agent 
The  supplier 

Completeness  of  innovation  package 

Implementation  delays 

Prior  need 

Command  support 

User  participation 

User  training 


General  social  conditions  in 
society 

General  political  conditions 

Critical  events,  crises,  and 
revolutions 

Acting  on  the  adopting 
organization 


aShields,  1976. 


Table  4 


Predictive  Model  of  Technology  Transfer  (The  Linker  Modex 
of  Jolly  and  Creighton,  1977) 


The  model  may  be  expressed  in  equation  form  such  that: 


Where 


L. 

i 


E0  C  +  6  C 
11  2  2 


*  +  +  0.C, 
D  k 


Li  =  Linker  index  for  an  organization  i 
9j  =  A  measure  of  factor  utilization,  0j  range  0  1 

=  A  measure  of  the  factor  contribution,  =  1 

Note:  The  linking  mechanism  necessary  to  achieve  effective  technology  trans¬ 
fer  is  described  by  identifying  the  factors  that  contribute  to  movement 
of  technology  from  the  source  of  knowledge  (supplier)  to  the  utiliza¬ 
tion  of  knowledge  (user/receiver) . 


3. 


Formal  Organization  of  the  User:  Refers  to  the  user's  perception 
of  his/her  formal  organization's  influence  on  the  use  of  technical 
information;  consider  infrastructure  elements,  such  as  the  power 
structure,  the  nature  of  the  business,  the  management  style,  the 
resources  available,  management's  attitudes,  the  amount  of  bureau¬ 
cratic  procedures,  and  the  stability  of  the  organization  (e.g., 
dependent  on  levels  of  users) . 

4.  Selection  Process  for  Projects  (User's  Contribution);  Refers  to 
the  user's  input  to  the  selection  of  the  R&D  project  (e.g.,  feed¬ 
back,  HRNs) . 


Informal  Factors 

5.  Capacity  of  the  Receiver  (User)  ;  Refers  to  the  individual  charac¬ 
teristics  associated  with  the  extent  of  use  of  a  training  product 
(e.g.,  traits,  attitudes  of  a  person,  such  as  leadership  qualities, 
educational  experience,  age,  social  status,  rank,  etc.). 

6.  Information  Linkers  in  the  Receiving  Organization;  The  person  or 
persons  who  promote  technology  transfer  process  (e.g.,  see  Appendix 
B,  such  as  sponsor's  representative,  commandant,  front-line  user, 
congressional  staffer) . 

7.  Credibility  as  Viewed  by  the  Receiver;  Refers  to  the  user's  per¬ 
ception  (assessment)  of  the  reliability  and  accuracy  of  the  informa¬ 
tion  (associated  with  the  training  product) .  Credibility  is  a 
function  of  the  perceived  reliability  and  accuracy  of  both  the 
source  and  the  channel  through  which  the  information  flows.  The 
extent  of  use  and  the  rate  of  adopting  research  output  correlates 
with  the  credibility  of  the  available  technology. 

8.  Perceived  Reward  to  the  User;  Refers  to  the  perceived  and  actual 
recognition  of  using  an  innovative  product  in  the  organization  of 
which  the  user  is  a  member ;  can  be  divided  into  two  broad  categories : 

(a)  intrinsic — opportunity  to  use  skills,  to  gain  new  knowledge,  to 
deal  with  challenging  problems,  and  to  have  freedom  to  follow  up 
one's  own  ideas  (akin  to  self-growth  in  competence  tendency); 

(b)  extrinsic — monetary  reward,  increased  administrative  authority, 
association  with  top  executives,  etc.  (It  is  suggested  later  that 
technology  transfer  in  the  Army  has  relied  more  on  the  intrinsic 
reward  system  than  on  the  extrinsic  reward  system.) 

9.  Willingness  To  Be  Helped:  Awareness,  or  familiarization,  of  an  in¬ 
novative  product  is  not  sufficient  to  insure  acceptance  and  use  of 
the  product  (e.g.,  Freda  &  Shields,  1979).  There  must  be  a  self- 
interest  (internal  motivation)  to  improve  one's  operational  setting 
via  the  use  of  new  training  procedures. 


The  Linker  Model's  main  contribution  is  the  recognition  of  both  procedural 
(formal)  and  behavioral  (informal)  factors  influencing  successful  dissemination 
of  information  about  a  training  product.  Jolly  (1975)  measured  these  factors 
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by  studying  the  responses  selected  organizations  made  to  questions  designed 
to  describe  each  factor.  Organizations  were  found  to  differ  statistically 
on  several  of  these  factors — organizations  defined  as  having  employed  ef¬ 
fective  technology  transfer  beforehand  differed  in  these  responses,  relative 
to  those  organizations  that  were  not  effective.  The  significance  of  this 
research  is  that  it  is  a  first,  but  essential,  step  in  predicting  the  suc¬ 
cess  of  technology  transfer.  One  issue  that  needs  to  be  resolved  with  this 
prediction  technique  is  the  development  of  a  reference  standard  (baseline) 
against  which  to  judge  technology  transfer  based  on  the  aforementioned  fac¬ 
tors  in  the  model. 


Institutionalize  the  Findings 

After  user  acceptance ,  it  is  important  to  note  the  differences  between 
the  initial  utilization  and  subsequent  implementation  of  the  training  product 
(Drucker,  1977) .  Utilization  refers  to  cooperative  efforts  between  the  re¬ 
searcher  and  user  to  incorporate  the  training  product  within  the  operational 
setting.  This  stage  can  involve  6.1  and  6.2  efforts  to  support  6.3A  success. 
Implementation  involves  further  use,  operational  adaptation,  and  expansion  of 
the  use  of  the  training  product.  This  activity  is  generally  performed  by 
the  users  above,  following  6.3A  efforts,  and  it  is  not  generally  a  formal 
function  of  human  resources  RDT&E. 

The  current  RDT&E  funding  categories  present  a  "Catch-22"  situation  for 
the  HR  RDT&E  community.  On  the  one  hand,  congressional  scrutiny  of  military 
training  budgets  forces  the  research  community  to  justify  its  existence  by 
demonstrating  effective  use  of  its  training  products.  On  a  research  level, 
this  goal  is  translated  into  developing  objective  measures  of  users'  benefits. 
On  the  other  hand,  the  fiscal  control  lent  to  the  HR  RDT&E  community  officially 
stops  at  tne  6.3A  funding  category  (nonsystem  advanced  development),  and  fur¬ 
ther  control  of  the  training  product  is  taken  over  or  dispersed  by  other  agen¬ 
cies  from  the  6.4  (engineering  development)  to  6.7  (operational  system  devel¬ 
opment)  funding  categories.  This  procedure  carries  over  from  weapon-systems 
development  practice,  irs  which  software  (e.g.,  training)  development  is  made 
equivalent  to  the  hardware  (e.g.,  materiel  system)  development  process.  [One 
aspect  of  this  analogy  is  the  difference  in  visibility,  for  justification 
purposes,  between  a  software  (nontangible)  product  and  a  hardware  (tangible) 
product.]  Therefore,  if  the  HR  RDT&E  community  is  to  be  held  accountable 
for  the  acceptance  and  effective  use  of  their  training  products,  it  would 
appear  reasonable  to  allocate  to  that  community  a  significant  amount  of 
budgetary  control  over  the  dissemination,  utilization,  and  implementation 
of  their  training  products.  At  present,  Congress  is  asking  the  HR  RDT&E 
community  to  be  responsible  for  the  operational  effectiveness  of  their  train¬ 
ing  products,  over  which  that  community  has  no  formal  budgetary  control. 

An  example  of  institutionalization  (Drucker,  1977)  is  a  personnel  test 
that  is  critically  validated  (utilization)  and,  after  the  user  receives  it, 
is  subsequently  standardized  on  a  larger  scale  (implementation) .  Another 
example  is  a  prototype  training  course  developed  for  a  particular  training 
manager  (utilization)  which  is  then  used  by  the  training  manager  for  wide¬ 
spread  application  in  a  number  of  training  locations  (implementation) ,  The 
final  stage  is  policy,  which  is  not  a  formal  part  of  the  HR  RDT&E  effort. 
During  the  policy  stage,  use  of  the  training  product  has  become  standard 
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practice,  defined  and  supported  by  Army  regulations.  When  this  happens, 
use  of  the  training  product  has  become  "routinized"  (Yin,  Quick,  Bateman, 
&  Marks,  1978).  The  stages  of  the  institutionalization  of  findings  are 
presented  in  Figure  6. 


Figure  6.  Institutionalize  the  findings. 


Yin  et  al.  (1978)  have  attempted  to  quantify  and  measure  the  stages  of 
the  institutionalization  of  an  innovative  product.  They  contend  that  an 
emphasis  on  this  activity  is  lacking  in  the  current  RDT&E  approach  to  tech¬ 
nology  transfer.  To  facilitate  the  analysis  of  the  institutionalization 
process,  the  process  of  eventual  routinization  is  viewed  as  a  composite  of 
both  passages  and  cycles.  A  passage  occurs  when  a  formal  transition  from 
one  organizational  state  to  another  has  been  accomplished.  For  example, 
establishing  new  procedures  for  instructional  development  constitutes  a  pas¬ 
sage  .  A  passage  reflects  increased  organizational  support  for  an  innovative 
product,  generally  occurs  only  once,  and  is  indicative  of  its  having  become 
a  more  integral  part  of  the  organization.  In  general,  the  more  passages  that 
have  been  achieved,  the  more  routinized  a  product  may  be  regarded.  A  cycle, 
on  the  other  hand,  is  an  organizational  event  that  occurs  repeatedly  during 
the  lifetime  of  an  organization.  In  general,  the  more  cycles  that  an  inno¬ 
vation  survives,  the  more  routinized  it  is.  Thus,  "cycle"  applies  to  repeated 
events  that  occur  as  part  of  an  organization's  operations  and  that  may  affect 
an  innovative  product. 

Table  5  presents  Yin's  breakdown  of  routinization  in  terms  of  resources 
and  operations  specific  to  passages  and  cycles.  Observation  or  use  of  these 
resources/operation  occurs  during  particular  stages  of  the  routinization 
process.  Similar  to  the  stages  of  the  institutionalization  process  are  the 
stages  of  the  "life  histories"  process  underlying  routinization.  These 
stages  are  improvisation  (utilization) ,  expansion  (implementation) ,  and 
disappearance  (policy) .  The  improvisation  stage  is  the  initial  period,  fol¬ 
lowing  user  acceptance,  during  which  the  product  is  used,  and  no  significant 
passages  or  cycles  need  occur  during  this  stage.  The  main  goal  here  is  to 
use  the  product  at  some  meaningful  levei/frequency  for  some  specified  period 
of  time.  The  expansion  stage  is  characterized  by  the  growing  use  of  the 
product  and  the  achievement  of  several  passages  and  cycles.  Finally,  the 
disappearance  stage  involves  the  completion  of  the  remaining  passages  and 
cycles.  During  this  stage,  the  product  continues  to  be  used  but  eventually 
loses  its  recognition  as  an  innovative  product — i.e.,  it  achieves  the  status 
of  standard  agency  practice  (policy) . 

Table  6  organizes  the  occurrence  of  passages  and  cycles,  along  with 
their  respective  resources/operations,  into  specific  stages  of  the  life 
history  of  an  innovative  product.  Table  7  presents  10  factors  associated 
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Table  7 


Factors  Associated  with  the  Facilitation  of  the 
Routinizaticn  of  an  Innovative  Product 


1.  Core  application  in  an  agency's  activities 

u  Determine  by  observing  if  innovative  product  has  lisplaced  some 
significant  function 

2.  Minimal  competition  for  resources  among  different  applications 

3.  Service  payoffs;  should  include 

•  Specific  nature 

•  Perceived  by  many  people 

•  Consensus  that  payoff  exists 

4.  Prior  need  for  innovative  product 

5.  Client-,  support 

6.  Community  support 

7.  Top  administrative  support  within  the  agency 

8.  Practitioner  support  within  the  agency 

9.  Active  innovator  support 

10.  No  adversary  group  that  specifically  opposes  the  innovative  product 
Source:  Yin  et  al.f  1978. 

Note:  ^actors  5,  6,  7,  and  8  can  be  used  in  relation  to  the  different  levels 
of  users  presented  in  Appendix  B, 


26 


with  the  facilitation  of  routinization  of  an  innovative  product  (Yin  et  al., 
1978)  . 

Table  8  displays  the  extent  of  influence  these  10  factors  had  on  routini 
zation  in  various  locations  using  different  innovations.  The  routinization 
process  is  graphically  portrayed  in  Figure  7. 

The  relevance  of  Yin's  approach  to  Army  training  technology  transfer 
resides  in  the  use  of  organizational  indicators  to  define  the  extent  that  a 
training  product  is  successfully  incorporated  within  an  operational  setting. 
Within  this  framewor!  ,  reliance  upon  indicators  such  as  Uwilization  reports 
(AR  70-8)  provides  only  a  partial  picture  of  where  the  product  stands  in  the 
way  of  incorporation.  Other  indicators  could  be  modified  and  incorporated 
by  Army  decisionmakers  into  an  "institutionalization  checklist,"  which  would 
provide  a  documented  metric  of  product  utilization. 


Monitoring,  Evaluation,  and  Feedback 


This  activity  is  ongoing  throughout  all  tne  main  activities  of  the  model 
For  example,  during  requirements  analysis,  emergent  requirements  from  the 
field  or  policy  makers  can  be  evaluated  and  possibly  incorporated  into  the 
formulation  of  ongoing  RDT&E  efforts.  During  RDTSE,  user-researcher  dialog 
can  result  in  review  of  training  design  respecification  for  possible  incor¬ 
poration  into  ongoing  R&D  efforts.  During  dissemination  and  institutionali¬ 
zation,  researcher  and  user  feedback  can  provide  information  to  guide  the 
acceptance  and  effective  use  of  a  training  product.  Models  predictive  of 
technology  transfer  are  also  used  to  measure  the  extent  of  successful  incor¬ 
poration  of  a  training  product  during  institutionalization.  Methodologies 
employed  in  this  activity  consist  of  multivariate  prediction  models,  on-site 
interviews,  and  monitoring  and  user  feedback  reports.  A  ma;jor  point  of  em¬ 
phasis  is  that  this  activity  should  be  continuous,  even  during  the  institu¬ 
tionalization  process.  Once  a  training  product  is  defined  as  standard  prac¬ 
tice  (policy  or  routinization) ,  monitoring  and  evaluation  of  this  practice 
can  result  in  a  new  emergent  requirement,  which  in  turn  initiates  the  train¬ 
ing  technology  transfer  process.  Notice  that,  as  previously  mentioned,  the 
training  technology  transfer  process  is  cyclical  in  nature,  and  only  linear 
for  explanatory  purposes. 


DISCUSSION  AND  SUGGESTIONS 

The  following  topics  p  jT&SGlVu  discussions  of  the  aforementioned  model , 
and  suggestions  to  improve  Army  training  technology  transfer. 


Organization  of  Regulations  and  Procedures 


The  procedures  defining  certain  aspects  of  Army  training  technology 
transfer  are  in  several  documents.  It  is  suggested  that  an  ARI  regulation 
could  consolidate  relevant  sections  of  Army  regulations  in  accordance  with 
the  model  presented  in  this  paper.  A  general  outline  is  presented  in  Table  9 
to  help  in  this  endeavor. 
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Factors  Hypothesized  to  Facilitate  Routinization  (Case  Studies) 


of  these  factors  is  indicated  by  a  plus  (+) 
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Essentially,  one  regulation  could  serve  as  a  system  concept  document 
whose  sections  are  defined  in  -erms  of  the  activities  of  the  model.  Specific 
parts  of  the  current  Army  regulations  can  be  incorporated  within  each  ac¬ 
tivity  of  the  model.  The  indicators  are  presented  in  Table  9  for  that  pur¬ 
pose.  It  is  assumed  that,  if  the  general  flow  of  the  model  is  accepted  as 
reasonable,  then  Army  decisionmakers  will  be  able  to  clarify  problems  in 
training  technology  transfer. 


A  Comprehensive  Requirements  Analysis 


Shields  (1976)  discussed  how  current  models  of  technology  transfer  do 
not  provide  adequate  information  (a)  to  insure  the  success  (or  improve  the 
chances  of  success)  of  the  implementation  process,  (b)  to  promote  the  time¬ 
liness  of  technology  transfer  (i.e.,  turnaround  time  from  what  is  wanted  to 
what  is  produced) ,  and  (c)  to  incorporate  a  wider  range  of  influential  fac¬ 
tors,  such  as  economic,  organizational,  and  political  considerations.  One 
way  to  achieve  these  objectives  is  to  weigh  systematically  and  exhaustively 
the  range  of  effect  (positive  and  negative  outcomes)  cf  a  proposed  training 
product  during  the  analysis  of  requirements  stage.  Such  a  procedure  is 
presently  lacking  in  training  technology  transfer.  An  attempt  to  accomplish 
this  task  has  been  initiated  by  the  Navy  Training  Analysis  and  Evaluation 
Group  (TAEG)  with  a  contract  to  IBM  to  develop  a  comprehensive  educational 
technology  assessment  model  (ETAM;  see  references  in  the  Analyse-  of  Require¬ 
ments  section  of  Appendix  C) .  The  ETAM  can  be  viewed  as  a  set  of  procedures 
to  evaluate  whether  a  proposed  trair ir>g  innovation  is  worth  the  subsequent 
RDT&E  effort.  Details  combining  .'vervi<  ’  and  steps  in  ETAM  are  presented 
in  Figure  8  and  Appendix  E.  The  ultimate  goal  of  ETAM  is  a  computer-based 
system  which  contains  (a)  taxonomic  information  of  major  classes  of  training 
products,  (b)  input  and  retrieval  capabilities  to  compare/contrast  the  pro¬ 
posed  innovation  with  those  already  contained  in  the  data  base,  and  (c)  .inter¬ 
active  routines  of  cost -benefit  analyses  using  both  qualitative  and  quantita¬ 
tive  weighting  factors  to  estimate  the  worth  of  the  training  product.  The 
relevance  of  ETAM  to  the  Army  training  technology  transfer  process  resides 
in  its  structure  for  a  comprehensive  requirements  analysis.  Army  decision¬ 
makers  might  find  ETAM's  flow  of  activities  helpful  in  defining  the  decision 
stages  of  user-researcher  interaction  so  as  to  facilitate  the  clarity  of  user 
needs,  translated  into  a  researchable  question.  The  main  point  here  is  the 
emphasis  on  clarity  of  problem  definition. 


A  Methodology  for  the  Dissemination  of  Information 

There  is  at  present  no  set  of  procedures  outlining  a  sequence  of  steps 
to  promote  the  acceptance  and  use  of  a  training  product.  Welsh  (1977)  has 
developed  a  methodology  which  may  resolve  some  of  the  problems  associated 
with  the  dissemination  of  information.  This  methodology  is  designed  to 
satisfy  user  requirements  through  the  systematic  dissemination  of  products. 

The  methodology  consists  of  10  primary  steps: 

1.  Planning, 

2.  Product  modification. 
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Figure  8.  ETAM  procedural  sequence. 


3.  Identifying  appropriate  diffusion  strategies/tactics, 

4.  Conducting  needs  assessments, 

5.  Focusing  upon  specific  target  audiences, 

6.  Identifying  early  adopters  and  opinion  leaders  within  targets 
audience, 

7.  Setting  forth  procedures  for  contacting  early  adopters  and  opinion 
leaders , 

8.  Utilizing  early  adopters  and  opinion  leaders  to  sustain  diffusion 
plan, 

9.  Systematically  evaluating  effects  of  a  product  upon  targeted  set¬ 
tings,  and 

10.  Systematically  evaluating  diffusion  strategies/tactics  in  relation 
to  product  impact  upon  targeted  audiences. 

Steps  1,  4,  and  9  pertain  to  conditions  for  change;  step  2  pertains  to  char¬ 
acteristics  of  the  innovation;  steps  5,  6,  and  7  pertain  to  characteristics 
of  the  target  audience;  and  steps  3,  7,  and  8  pertain  to  linkage  systems. 

Only  characteristics  of  the  innovator  are  not  addressed  by  the  methodology 
(see  Wolf,  1977,  for  description  of  linkage  agents'  training  program).  Ap¬ 
pendix  F  describes  Welsh’s  methodology  in  more  detail. 

The  importance  of  this  methodology  for  Army  training  technology  transfer 
is  the  specificity  of  guidelines  to  promote  user  acceptance  of  training  prod¬ 
ucts.  It  is  plausible  that  Array  decisionmakers  could  consider  the  use  or 
modification  of  this  methodology  as  a  first  step  in  understanding  the  dis¬ 
semination  process. 


Documenting  the  Institutionalization  of  a  Training  Product 


Although  current  funding  control  of  Amy  training  technology  transfer 
stops  at  the  6.3A  level  (user  acceptance),  congressional  concern  still  focuses 
on  the  accountability  of  human  resources  RDT&E  to  develop  and  produce  objec¬ 
tive  measures  of  user  benefits.  One  aspect  of  this  concern  is  to  document 
the  flow  of  activities  during  the  institutionalization  of  a  training  product. 
It  is  suggested  that  one  way  to  do  this  is  to  use  or  modify  the  passage  and 
cycle  framework  developed  by  Yin  et  al.  (1978).  By  documenting  those  passages 
and  cycles  that  occur  during  the  utilization,  implementation,  and  policy 
stages,  Amy  decisionmakers  can  determine  the  extent  of  incorporation  of  a 
successful  or  unsuccessful  training  product.  Table  10  presents  a  proposed 
classification  scheme  to  document  activities  during  the  institutionalization 
of  a  training  product.  This  scheme  could  be  incorporated  into  an  ARI  regula¬ 
tion,  consolidating  infomation  with  respect  to  Army  training  technology 
transfer. 


Recurrent  Factors  to  Consider  During  Training  Technology  Transfer 


Two  major  classes  of  factors  recurrent  throughout  the  whole  training 
technology  transfer  process  are  sponsorship  and  self-renewal  capability. 

Sponsorship.  The  authoritative  support  and/or  active  proponent  of  a 
training  product  and  necessary  considerations  influencing  the  successful 
incorporation  of  an  innovation  sponsorship  may  be  viewed  on  two  levels. 

1.  User.  Initially,  highly  authoritative  policy  makers  are  needed  to 
promote  the  technology  transfer  process.  Researcher  interaction  with  all 
levels  of  users  should  follow,  however,  to  insure  the  endurance  of  this 
prjcess  to  completion  (i.e.,  acceptance  and  effective  use  of  the  training 
product  by  the  user) .  That  is ,  successful  incorporation  of  the  training 
product  will  depend  more  on  within-unit  sponsorship  as  the  technology  trans¬ 
fer  process  proceeds.  At  the  outset  of  a  project,  active  proponents  should 
be  sought  and  encouraged,  such  as  a  sponsor's  representative  who  would  work 
closely  with  the  researcher  throughout  all  the  activities  of  the  training 
technology  transfer  process.  This  linkage  agent  role  for  the  user  would 
help  to  minimize  current  and  subsequent  resistance  from  the  eventual  "front¬ 
line"  user  (i.e.,  those  at  the  implementation  and  receiver  levels  of  users). 

For  example,  the  sponsor's  representative  could  be  given  a  temporary-duty 
status  or  assigned  "contact"  days  to  work  and  consult  "in-house"  with  the 
researcher.  During  this  arrangement,  the  representative  could  deliver  or 
arrange  seminars  designed  to  explain  the  training  product  to  prospective 
user  agencies.  Pros  and  cons  could  be  discussed  at  these  meetings,  details 
negotiated,  and  feedback  provided  for  incorporation  into  the  RDT&E  activity. 
Thus,  user  resistance  encountered  during  these  sessions  would  be  recognized, 
discussed,  and  perhaps  resolved  prior  to  full-scale  dissemination  of  the 
training  product.  Therefore,  the  sponsor's  representative  would  act  as  a 
coordinating  influence,  translating  "front-line"  user  concerns  into  useful 
RDT&E  considerations. 

2.  Researcher .  It  is  the  researcher's  responsibility  to  determine  the 
technical  feasibility  of  satisfying  the  user's  requirements.  Sponsorship 
within  this  framework  focuses  on  the  RDT&E  support  given  to  a  training  prod¬ 
uct.  Collocation  of  researchers  with  users,  such  as  ARI  field  unius,  is  one 
way  in  which  this  support  can  be  perceived  more  quickly  by  the  user.  It  is 
suggested,  however,  that  the  role  of  field  units  be  expanded  to  include  re¬ 
sponsiveness  to  regional  (vs.  local)  Army  needs,  with  major  and  minor  research 
efforts  allocated  jointly  by  the  field  unit  and  ARI  Headquarters.  Similar 
efforts  have  been  demonstrated  by  the  Navy's  Field  Engineering  Offices,  which 
have  resulted  in  greater  timeliness  in  response  to  user  requests  for  innova¬ 
tions.  It  is  realized  that,  in  the  extreme,  collocation  could  result  in 
parochialism  and  unresponsiveness  to  the  global  needs  of  the  Army  (Sands  & 
Glaser,  1978).  However,  with  these  considerations  in  mind,  the  lag  in  im¬ 
plementation  of  training  technology  R&D  findings  has  been  shortened  when 
RDT&E  has  been  collocated  with  the  user  (Alluisi,  1977). 

Self-Renewal  Capability.  This  factor  refers  to  those  variables  which 
help  the  user  to  incorporate  and  maintain  the  training  product  after  researcher 
intervention.  Several  variables  are  relevant  here: 
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1.  Training.  Introduction  to  and  on-the-job  training  in  the  use  of 
the  training  product  for  both  administration  and  implementation  levels  of 
users  should  be  initiated  no  later  than  the  dissemination  of  information 
activity.  Effective  training  programs  should  be  coordinated  with  the  re¬ 
searcher  and  user  (i.e.,  at  this  stage,  the  training  schools  would  be  con¬ 
sidered  a  "user"  also) ,  with  the  ultimate  goal  of  within-agency  (in-house) 
training  programs  that  would  prepare  subsequent  new  personnel  for  using  the 
training  product.  In-house  training  programs,  thus,  promote  both  initial 
competency  as  well  as  endurance  of  the  use  of  the  training  product. 

2.  Resistance.  As  implied  earlier,  resistance  to  a  novel  training 
product  is  a  normal  phenomenon.  However,  resistance  may  also  be  viewed  as 
an  indication  of  a  lack  of  researcher -user  cooperation  to  prepare  "front¬ 
line"  users  for  the  outcomes  of  the  training  product.  Resistance  can  be 
lessened  by  (a)  early  researcher-user  involvement  in  the  training  technology 
transfer  process,  (b)  the  sponsor’s  representative  serving  in  an  active 
linkage  agent  role  (i.e.,  marketing  and  persuading  prospective  agency  users), 
and  (c)  suitable  training  prior  to  the  incorporation  of  the  training  product 
within  a  unit. 

3.  Money .  Budgetary  considerations  should  be  planned  and  evaluated 
during  the  analysis  of  requirements  activity.  Fiscal  obligation  for  sup¬ 
porting  the  institutionalization  of  the  training  product  requires  strong 
sponsorship  at  the  policy  and  administration  levels  of  the  user.  There  is 
no  use  in  directing  efforts  toward  user  acceptance  if  subsequent  institu¬ 
tionalization  of  the  training  product  will  not  be  funded. 

4.  Personnel  Turnover.  This  concern  is  important  in  the  Army  because 
of  frequent  tour-of-duty  changes  (every  2  to  4  years) .  As  mentioned  earlier, 
a  good  in-house  training  program  should  lessen  personnel  turnover  problems 

in  maintaining  the  use  and  incorporation  of  a  training  product.  Also  sug¬ 
gested  is  an  extended  tour  of  duty  for  those  key  personnel  involved  in  the 
initial  utilization  of  a  high-priority  training  product.  Such  an  extension 
may  provide  the  necessary  foundation  for  the  subsequent  implementation  and 
standardization  of  the  training  product  within  the  unit. 

5.  Reward  (Incentive)  System.  A  set  of  procedures  (regulations)  should 
be  instituted  that  provides  incentives  for  researcher-user  efforts  in  train¬ 
ing  technology  transfer.  For  example,  both  implementation  and  receiver  levels 
of  the  user  could  be  given  official  recognition/awards,  cited  in  their  per¬ 
sonnel  records  for  promoting/using  the  training  product  in  an  effective  manner. 
Both  policy  and  administrative  levels  of  the  user  could  be  given  more  budget¬ 
ary  and  policy  leeway  to  promote  training  technology  transfer  (i.e.,  such  as 
being  allowed  to  spend  money  saved  in  implementing  the  training  product,  in 
other  areas  given  high  priority  by  the  local  user  agency) .  Other  possible 
incentives  are  accelerated  promotions  and/or  seniority,  choice  of  tour  of 
duty,  choice  of  training  programs,  or  monetary  bonuses. 


Possibilities  for  an  Evaluation  Scheme 


As  presented  earlier,  monitoring,  evaluating,  and  feedback  occur  through¬ 
out  the  entire  training  technology  transfer  process.  These  activities  may 
be  in  the  form  of  emergent  requirements ,  changes  in  spec: fications  during  the 
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RDT&E  effort,  validation  feedback  reports,  or  research  utilization  predic¬ 
tion  models.  If  given  the  funding  controls,  follow-up  feedback  evaluations 
should  also  be  conducted  by  the  researcher  (outside-agency)  on  an  annual 
basis,  while  the  user  (wichin-agency)  should  conduct  in-house  evaluations 
more  frequently  (i.e.,  semiannually).  The  specific  ways  of  conducting  such 
an  evaluation  are  lacking;  however,  the  following  format  is  suggested  for 
use . 


To  answer  definitively  the  question  of  how  many  training  products  are 
in  fact  used,  a  systematic  approach  must  be  formulated.  First,  the  types 
of  training  products  to  be  evaluated  must  be  defined.  One  could  start  by 
using  the  classification  scheme  reported  by  Drucker  (1977).  Second,  the 
events  that  accompany  the  acceptance  and  use  of  a  training  product  must  be 
clarified  and  documented  for  data  collection  purposes.  The  passages  and 
cycles  classified  under  the  stages  of  the  institutionalization  activity 
(Yin  et  al.,  1978)  could  serve  initially  as  the  critical  "utilization" 
events.  Both  of  these  suggestions  are  presented  together  in  matrix  format 
in  Table  11. 

If  this  scheme  is  used,  certain  passages  and/or  cycles  may  not  be  appropri¬ 
ate  to  certain  training  products,  and  thus,  modifications  will  have  to  be 
made  on  an  individual  basis.  This  kind  of  approach  would  provide  systemati¬ 
cally  obtained  information  on  whether  or  not  a  utilization  problem  exists,  and, 
if  so,  the  degree  to  which  this  problem  extends  into  the  institutionalization 
activity,  and  for  what  products.  Additional  information  needed  to  address 
this  problem  is  proposed  in  Appendix  G. 


A  Proposed  Modification  of  Current  RDT&E  Funding  Categories 


As  mentioned  previously,  materiel  system  RDT&E  categories  are  used  in 
training  research  RDT&E.  These  categories  essentially  direct  the  flow  of, 
and  diffuse  responsibility  for,  training  technology  transfer.  Unfortunately, 
neither  near-term  nor  long-term  training  R&D  efforts  can  be  isolated  into 
discrete  operational  control  afforded  by  the  current  RDT&E  categories.  Even 
well-planned  training  research  involves  (within-category)  serendipitous  find¬ 
ings  which  promote  reflection  on  and  modification  of  ongoing  research  efforts. 
It  is  suggested,  therefore,  that  it  would  be  desirable  to  modify  the  present  . 
funding  categories.  The  objective  of  this  proposal  is  twofold;  (a)  to  pro¬ 
vide  some  degree  of  fiscal  and  operational  control  by  the  research  community 
over  the  institutionalization  of  training  products,  thus  providing  the  re¬ 
search  community  with  the  capability  of  sharing  the  accountability  for  the 
acceptance  and  effective  use  of  the  training  product,  and  (b)  to  provide 
timely  end  well-planned  training  research  programs  that  support  Army  decision¬ 
makers  for  both  near-term  and  long-term  problems. 

Specifically,  this  proposal  suggests  an  incorporation  of  6. 1-6. 7  funding 
categories  into  two  research  funding  activities  administered  by  ARI .  This 
suggestion  is  outlined  in  Table  12.  The  primary  proposed  modification  to 
the  present  RDT&E  funding  system,  as  stated  above,  is  the  full-scale  opera¬ 
tional  and  budgetary  control  inherent  in  the  Programmed  Fund.  This  control 
promotes  acceptance  and  effective  use  of  a  training  product.  The  Programmed 
Fund  is  designed  to  provide  a  well-planned  effort  to  improve  training  tech¬ 
nology  transfer.  This  effort  is  realized  in  an  "Implementation  Plan"  v/hich 
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Proposed  Classification  Scheme  To  Measure  "Product  Utilization 
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passages,  C  =  cycles;  specific  descriptions  extracted  from  Yin  et  al.,  1978 


Table  12 


Proposed  ARI  Research  Funding  Activities 


Analytic  fund 


Programed  fund 


(30%  -  35%  budget) 

1.  Combines  previous  6.1  and  most 
6.2  monies 


2.  Research  directed  toward  provid¬ 
ing  input  for  both  near-term 
and  long-term  problems 

3.  Research  initiated  or  provided 

by:  (a)  spinoff  ideas  from 

programmed  fund,  (b)  contracts 
(basic  research) ,  (c)  in-house 
generation  (ILIR) ,  and  (d)  in- 
house  problem  definition  re¬ 
search  for  Army  training  (some 
previous  6.2  work). 


(65%  -  75%  budget) 

1.  Combines  some  6.2,  all  6.3A, 
and  adds  (as  appropriate) 

6. 4-6. 7  money  for  extending 
operational  control  to 
institutionalization 

2.  Research  directed  toward  solv¬ 
ing  mainly  near-term  problems 
(5  years) 

3.  Research  initiated  from  the 
top  down  (by  management) ,  or 
laterally  (from  the  field) . 
Research  funding  requires  an 
"Implementation  Plan,"  in 
which  researchers  and  users 
document  the  accountability, 
audit  trails,  and  evaluation 
schemes  for  each  of  the  follow¬ 
ing  activities:  (a)  require¬ 
ments  analysis,  (b)  RDT&E 
solutions,  (c)  dissemination  of 
information ,  and 

(d)  institutionalization. 


constitutes  much  "front-end"  analysis  prior  to  RDT&E  activity.  If  prior 
consideration  is  given  to,  and  allowed  for,  dissemination  and  institution¬ 
alization  problems  (given  adequate  funding  control) ,  increased  likelihood 
of  the  acceptance  and  use  of  the  training  products  may  result.  Appendix  H 
provides  a  suggested  format  for  an  implementation  plan  to  guide  RDT&E  efforts 
in  the  Programmed  Fund.  Finally,  the  Analytic  Fund  would  provide  both  the 
technical  data  base  and  problem-definition  guidance  for  supporting  research 
planning  in  the  Programmed  Fund.  Whereas  the  Programmed  Fund  would  see  a 
training  research  project  through  the  requirements  analysis,  RDT&E,  dissemi¬ 
nation,  and  institutionalization  activities,  the  Analytic  Fund  would  operate 
primarily  within  the  RDT&E  domain.  It  is  hoped  that  this  type  of  funding 
framework  will  provide  greater  researcher  involvement  in  the  control  over 
the  acceptance  and  use  of  a  training  product.  Given  the  funding  authority, 
the  researcher  may  then  feel  somewhat  more  justified  in  being  held  account¬ 
able  for  obtaining  objective  measures  of  user  benefits. 


SYNOPSIS  OF  ISSUES  IN  TECHNOLOGY  TRANSFER 
Definitional  Issues  in  General 


1.  Technology  transfer  is  a  process  by  which  existing  research  knowledge 
is  transferred  operationally  into  useful  processes,  products,  or  programs 
that  fulfill  actual  or  potential  public  or  private  needs.  The  user  can  be 
defined  in  two  dimensions:  organisational  sector  (civilian  and  military), 
and  the  functional  level  of  involvement  in  the  Army  technology  transfer 
process  (scientific/informative  review,  policy,  administration,  implementa¬ 
tion,  and  receiver).  These  dimensions  are  depicted  in  a  User  Identification 
Matrix  (see  Appendix  B) . 


Modeling  and  Transfer  in  General 


1.  Development  of  a  systems  model  a^d  organization  of  regulations  to 
reflect  the  flow  of  technology  transfer  within  an  agency.  Four  major  activi¬ 
ties  constitute  or  define  human  resources  technology  transfer  in  the  Army: 

•  Analysis  of  requirements  (e.g.,  needs  assessment,  resulting  in  a 
researchable  question) ; 

•  Research,  development,  testing,  and  evaluation  of  solutions  (RDT&E, 
resulting  in  a  research  product) ; 

•  Dissemination  of  findings  (can  result  in  user  acceptance) ;  and 

•  Institutionalization  (starts  with  the  utilization  of  the  product  by 
the  user  and  eventually  is  incorporated  within  the  user's  agency  as 
a  policy  matter) . 

2.  Between-  Versus  Within-Technology  Transfer  (see  Chapter  1). 
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Specific  Needs  Assessment 


1.  Lack  of  systematic  methodology  to  assess  the  needs  of  the  military 
user  prior  to  the  initiation  of  R&D.  Needs  assessment  should  consider 

(a)  researcher-user  interaction  to  arrive  at  a  consensually  defined  research- 
able  question,  (b)  program  analysis  techniques  to  define  and  weigh  alternat¬ 
ing  research  approaches,  (c)  cost-benefit  analysis  to  include  subjective 
estimates  of  utility  of  a  particular  research  proposal,  and  (d)  individual, 
innovation,  organizational,  political,  and  economic  variables  which  would 
influence  the  formulation  of  an  R&D  plan. 

2.  The  extent:  to  which  dissemination  and  R&D  implementation  plans  should 
be  incorporated  in  needs  assessment.  Assuming  that  future  funding  for  a 
particular  R&D  product  is  predictable,  one  could  plan,  prior  tc  RDT&E ,  how 

to  disseminate  information  about  the  R&D  plan,  as  well  as  how  to  transfer 
responsibility  from  the  researcher  to  the  user  in  line  with  the  shift  of 
operational  funding  control  during  institutionalization  of  the  R&D  product 
(e.g.,  shifting  from  6.3  to  6. 4-6. 7  funds).  The  analogy  is  planned  compari¬ 
sons  rather  than  post  hoc  analysis. 


Specific  RDT&E 


How  appropriate  are  the  current  funding  categories/processes  (e.g., 

6. 1-6. 7)  for  human  resources  R&D  activity?  It  may  be  desirable  to  consider 
modification  of  the  present  funding  categories.  Specifically,  this  proposal 
suggests  an  incorporation  of  6. 1-6. 7  funding  categories  into  two  research 
funding  activities  (see  Table  12) . 


Dissemination 


1.  Procedures  and  factors  involved  in  dissemination  of  information 
about  a  research  product.  Dissemination  occurs  after  RDT&E  and  ends  in  user 
acceptance . 

2.  Classification  and  documentation  of  the  events  surrounding  the 
transition  between  researcher  responsibility  and  user  institutionalization 
of  a  research  product.  Congressional  scrutinizing  of  military  training  bud¬ 
gets  makes  the  research  community  justify  their  existence  by  demonstrating 
effective  use  of  their  training  products  by  the  target  user.  On  a  research 
level,  this  goal  is  translated  into  developing  objective  measures  of  user's 
benefits.  On  the  other  hand,  however,  the  amount  of  fiscal  control  lent  to 
the  human  resources  R&D  community  generally  stops  at  the  6.3  funding  category 
officially,  and  further  control  of  the  research  product  is  taken  over  or  dis¬ 
persed  by  other  agencies  from  the  6.4  to  the  6.7  funding  categories.  This 
procedure  is  a  carry  over  from  weapon-systems'  development  practice  in  which 
software  (e.g.,  training)  development  is  made  equivalent  to  the  hardware 
(e.g.,  materiel  system)  development  process.  Therefore,  if  the  human  re¬ 
sources  R&D  community  is  to  be  held  accountable  for  the  acceptance  and  ef¬ 
fective  use  of  their  training  products,  a  significant  amount  of  budgetary 
control  over  the  dissemination,  utilization,  and  implementation  of  their 
training  products  should  be  allocated  to  the  HR  RDT&E  community.  At  present 
Congress  is  asking  the  HR  RDT&E  community  to  be  responsible  for  the  operational 
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effectiveness  of  their  training  products  over  which  the  human  resources  R&D 
community  has  no  formal  budgetary  control. 

3.  Linkage  agents  and  change  agentry  effects  on  user  acceptance  of 
research  products.  A  linkage  agent  role  of  the  sponsor's  representative 
would  help  to  minimize  current  and  subsequent  resistance  from  the  eventual 
"front-line"  user — those  at  the  implementation  and  receiver  levels  of  users 
{e.g.,  trainers  and  students). 


Institutionalization 


1.  Documentation  of  the  critical  events  during  the  institutionaliza¬ 
tion  of  a  research  product. 

2.  Determining  the  "time  line"  of  a  research  product's  incorporation 
by  a  military  user. 


Prediction,  Monitoring,  Evaluation,  and  Feedback 


1.  Predicting  the  successful  incorporation  of  a  research  product  by 
the  user.  Several  models  have  been  developed  to  predict  the  outcome  of  a 
research  product;  this  approach  attempts  to  translate  descriptive  factors 
into  numerical  values  that  can  be  incorporated  into  a  mathematical  equation, 
whose  result  would  provide  an  index  representative  of  the  predicted  success/ 
failure  of  the  acceptance/use  of  a  research  product  by  the  user.  The  Linker 
Model's  main  contribution  is  the  recognition  of  both  procedural  (formal)  and 
behavioral  (informal)  factors  influencing  successful  dissemination  of  infor¬ 
mation  about  a  research  product. 

2.  Developing  a  prediction  methodology  in-house.  To  answer  definitively 
the  question  of  how  many  research  products  are  actually  used,  a  systematic 
approach  must  be  formulated  (see  Appendix  G) . 

3.  Interorganizational/agency  involvement.  The  information  depicting 
those  DOD  agencies  that  spend  RDT&E  funds,  and  the  types  of  RDT&E  funds  used 
is  a  matrix  consisting  of  6.1  through  6.7  funds  listed  along  one  dimension, 
and  the  names  of  the  agencies  along  the  other  dimension.  The  question  of 
interest  is  from-whom-to-whom  are  funds  handed  off  during  the  transition  of 
RDT&E  funding  control  of  the  research  product.  This  information  would  indi¬ 
cate  the  extent  of  desirable  interaction  with  specific  agencies  involved  in 
the  nandoff  of  budgetary  and  operational  control  of  the  research  product. 

(For  example,  one  could  see  how  many  agencies  spend/use  funds  during  the 

6. 3-6.4  interval.)  This  matrix  would  serve  as  a  "funding  flow"  chart  con¬ 
sisting  of  clusters  of  agencies  which  may  be  correlated  with  organizational 
events  critical  to  the  outcome  of  the  research  product. 

4.  Determining  baseline  success  rates.  Success  rates  of  research  prod: 
uct  utilization  would  help  to  guide  subsequent  funding  and  research  directions. 
Determining  the  baseline  rate  has  not  been  addressed  empirically.  The  defi¬ 
nition  issue  could  be  resolved  by  using  both  objective  and  subjective  sources 
of  information,  as  mentioned  previously.  Then  the  scope,  to  be  addressed, 
could  be  limited  up  to  6.3  (user  acceptance) ,  or  could  include  the  transition 
period  (6. 3-6. 4  interval)  of  budgetary  and  operational  control. 
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5.  Monitoring,  evaluation,  and  feedback.  This  activity  goes  on  during 
all  the  main  activities  of  the  model.  Once  a  research  product  is  defined  as 
standard  practice  (policy  or  rout xnization) ,  monitoring  and  evaluation  of 
this  practice  can  result  in  a  new  emergent  requirement,  which  in  turn  initi¬ 
ates  the  training  technology  tj'ansfer  process. 


Recurrent  Issues 

Recurrent  througho’.c  the  training  technology  transfer  process  are  spon¬ 
sorship  and  self-renewal  capability;  incorporation  of  an  innovation  sponsor¬ 
ship  may  be  viewed  on  two  levels:  user  and  researcher. 
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APPENDIX  A 


DOCUMENTATION  OF  INFORMAL  COMMENTS  AND  ANECDOTAL  EVIDENCE 


Document  1  (Shields,  1976) 

1.  In  one  unit,  in  the  FORCES  COMMAND  (FORSCOM)  SCOPES  (an  engagement 
simulation  system  for  the  infantry)  was  introduced,  briefly  used, 
and  for  the  past  year  has  simply  sat  idle. 

2 .  The  Training  Aids  and  Audiovisual  Support  Services  Organization 
(TASO)  at  another  FORSCOM  installation  indicated  that  a  game  de¬ 
signed  to  assist  tankers  in  learning  combat  tactics  had  not  been 
checked  out  since  its  arrival. 

3.  A  tank-driver  trainer-simulator  was  found  sitting  in  a  warehouse — 
simply  occupying  a  great  deal  of  space — not  training. 


Document  2  (Sands  &  Glaser,  1978) 


1.  Army  Case  Study — The  Value  of  Stereoscopic 
Viewing  in  Image  Interpretation 

The  research  objective  was  to  assess  the  usefulness  of  stereoscopic  view¬ 
ing  within  serial  surveillance  systems  in  terms  of  quality  of  information  ob¬ 
tained  and  the  rate  at  which  it  is  extracted.  Both  tactical  and  strategic 
types  of  interpretations  were  used  in  setting  up  performance  measures  which 
were  administered  to  two  matched  groups  of  image  interpreters.  For  each 
measure,  stereo  pairs  were  provided  to  one  of  the  two  groups  and  nonstereo 
photographs  to  the  other.  Data  were  analyzed  by  comparing  mean  scores  through 
t-tests  and  analysis  of  variance. 

Stereo  viewing  and  nonstereo  viewing  of  the  tactical  and  strategic  photo¬ 
graphs  were  found  to  be  equivalent  in  terms  of  the  quality  of  information  pro¬ 
vided  and  confidence  expressed  by  interpreters  in  the  information  they  ex¬ 
tracted.  Accuracy  and  number  of  targets  identified  under  the  two  methods  of 
viewing  were  similar;  that  is,  no  statistically  significant  differences  were 
found . 

This  research  was  suggested  by  Army  researchers  after  they  had  been 
called  in  by  the  Air  Force  to  consult  on  a  similar  problem.  Findings  in  the 
Army  research  then  suggested  strongly  that  the  value  of  stereo  viewing  should 
not  be  taken  for  granted,  and,  in  fact,  led  Army  researchers  to  suggest  that 
the  need  for  the  stereo  capability  should  be  clearly  demonstrated  before  new 
display  equipment  with  stereo  capability  is  developed  for  use  of  interpreters 
in  detecting  and  identifying  militarily  significant  objects.  Several  other 
research  efforts  by  other  organizations  corroborated  these  findings. 


57 


But  Army  researchers  found  a  strong  existing  conviction  in  favor  of 
stereo  and  a  system  already  highly  geared  up  for  obtaining  and  interpreting 
stereo  imagery.  The  research  had  little  impact  on  decisions  to  build  more 
stereo  capability,  despite  consistent  replication  of  the  above  and  the  con¬ 
duct  of  several  projects  on  overlapping  imagery  (60%  is  needed  for  stereo) 
that  indicated  conditions  of  nonoverlap  (except  the  small  percent  needed  to 
insure  100%  coverage)  to  be  superior  in  time  to  interpret  and  no  different 
in  terms  of  accuracy  or  completeness  of  interpretation. 

The  practical  aspects  of  stereo  coverage  argue  strongly  against  it.  In 
addition  to  the  extra  30%  to  40%  in  time  required  for  stereo  viewing  by  the 
interpreter  mentioned  above,  there  are  several  severe  systems  costs  involved 
It  takes  twice  as  many  photographs  (and  processing)  to  cover  a  given  area 
using  stereo.  Dollar  cost  as  a  function  of  the  number  of  systems  using 
stereo  is  readily  derived.  It  also  means  that  twice  as  many  reconnaissance 
missions  need  to  be  flown  (again  one  can  compute  dollar  costs)  and  twice  as 
many  planes  and  lives  risked. 

And  still  systems  users  have  persisted  in  using  stereo. 


2.  Army  Case  Study — REALTRAIN 

REALTRAIN  is  an  improved,  lcw-cost  training  and  evaluation  technique 
for  use  in  Army  tactical  training  exercises  for  combat  units.  Realistic, 
two-sided  free-play  tactical  training  employing  recognized  principles  cf 
learning  is  achieved  through  simulated  combat  engagements. 

Originally  this  research  sought  to  develop  a  method  for  evaluating 
individual  tactical  performance  under  simulated  battlefield  conditions.  It 
was  felt  necessary  first  to  construct  job  situations  that  would  demand  that 
a  man  act  as  he  would  be  expected  to  act  on  a  battlefield.  It  was,  in  retro 
spect,  not  surprising  that  the  environment  developed  for  testing  became  a 
powerful  vehicle  for  training.  The  rationale  which  provided  the  basis  for 
the  initial  development  of  methods  for  simulating  the  combat  environment 
with  a  high  degree  of  psychological  fidelity  led  directly  to  the  REALTRAIN 
method  for  tactical  training. 

Before  a  research  organization  recommends  implementation  of  a  new  train 
ing  method  it  is  accepted  practice  to  determine  empirically  whether  the  new 
method  works  and  how  well  it  works  (the  degree  to  which  training  objectives 
are  achieved,  the  nature  of  the  skill  acquisition  curve)  and  to  compare  the 
new  method  with  the  method  it  was  designed  to  replace.  This  was  not  done 
with  REALTRAIN.  The  decision  was  made  by  the  Training  and  Doctrine  Command 
(TRADOC)  to  implement  the  method  before  the  standard  validation  procedures 
had  been  conducted. 

The  reasons  for  TRADOC' s  decision  were  (a)  the  heavy  cost  of  conducting 
an  evaluation  of  a  new  unit  training  technique  in  the  field,  (b)  the  rapid 
and  enthusiastic  acceptance  of  the  method  by  troops  and  commanders,  (c)  the 
overwhelming  face  validity,  and  (d)  the  fact  that  no  technique  for  realistic 
tactical  training  had  previously  existed. 


The  REALTRAIN  training  method  was  implemented  by  a  TRADOC  Mobile  Train¬ 
ing  Team  (MTT)  during  the  period  3  November  1S75  to  5  March  1976  at  four 
divisional  training  sites  throughout  the  U.S.  Army  Europe  (USAREUR) .  The 
implementation  in  USAREUR  afforded  an  opportunity  to  conduct  research  which 
could  be  used  to  improve  tactical  training  and  evaluation  techniques  further 
in  an  engagement  simulation  context,  specifically  providing  a  valuable  em¬ 
pirical  base  and  data  source  for  the  analysis  of  tactical  performance  by 
participants  in  the  exercise ,  participant  and  controller  reactions  to  this 
new  method,  and  the  cost  of  conducting  such  exercises. 

That  implementation  by  TRADOC  involved  a  cost  of  several  million  dol¬ 
lars.  REALTRAIN  is  still  not  being  effectively  utilized  today  for  a  number 
of  reasons: 

1.  REALTRAIN  has  entailed  very  significant — even  revolutionary — 
changes  from  the  ways  of  past  training  and  it  was  almost  too  big 
a  challenge  to  do  it  well. 

2.  One  specific  resource  requirement  that  has  given  heartburn  has 
been  the  requirement  for  controllers  who  are  required  for  conduct 
of  an  exercise,  though  controllers  learn  as  much  as  the  trainees. 

3.  Young,  inexperienced  officers  don't  like  to  conduct  exercises  and 
lose  badly,  as  many  of  them  do,  even  though  the  learning  experi¬ 
ences  are  invaluable.  By  the  same  token,  to  be  a  participant- 
casualty  through  inappropriate  performance  in  the  exercise  is 
also  stigmatic.  To  help  solve  the  young  officer  problem,  a  leader 
board  game  has  been  devised  to  get  the  officers  better  prepared 
for  the  exercises. 

4.  Logistical  requirements  are  overwhelming — training  ammo  is  expen¬ 
sive,  tactical  radios  to  support  control  of  the  exercise  are  hard 
to  get. 

TRADOC  officials  have  been  much  aware  that  initial  successful  utiliza¬ 
tion  of  REALTRAIN  may  rest  critically  upon  keeping  researchers  involved  in 
the  handing-off  process  to  help  solve  some  of  the  problems  of  utilization. 
The  purpose  of  a  new  program  is  to  do  just  that — have  researchers  assist  in 
preparing  a  method  for  implementation  and  observe  problems  of  utilization 
and  help  make  refinements,  periodically  return  to  reassess  utilization 
procedures . 

Comment:  Normally  the  early  grabbing  of  a  research  product  for  imple¬ 
mentation  before  the  completion  of  research  is  considered  in  the  nature  of 
eating  the  bean  sprouts,  intended  for  planting  to  ease  famine.  In  this 
instance,  however,  researchers  did  not  consider  that  harm  had  been  done  to 
the  ultimate  utilization  of  this  product,  especially  in  view  of  the  fact 
that  implementation  aided  research. 


3.  Army  Case  Study — Armor  School 


The  Armor  School,  in  conjunction  with  Naval  Training  Equipment  Center 
(NTEC) ,  contracted  with  General  Dynamics  to  develop  a  Miniature  Armor 
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Battlefield  (MAB).  The  MAB  had  radio-controlled  tanks  on  a  miniature  (6‘) 
field,  with  TV  sensors  in  the  tanks  and  hit/kill  sensors;  it  was  intended 
for  the  simulation  of  platoon  versus  platoon  engagements.  Development  cost 
was  approximately  $1  million. 

The  Armor  School  asked  HumRRO  to  evaluate  the  system  and  to  work  out 
training  procedures.  The  research  group  had  been  working  on  a  simplified 
versi-n  of  the  system  which  eliminated  some  problems  encountered  with  the 
TV  sensors  on  the  General  Dynamics  device.  The  evaluation  was  performed 
with  a  24-nour  field  exercise  as  the  performance  criterion;  the  system  was 
shown  to  be  effective  for  training  tank  crews.  A  follow-up  in  Europe  using 
commanders’  ratings  as  a  criterion  resulted  xu  higher  ratings  for  the  sys¬ 
tem  than  for  conventional  training  techniques. 

The  Armor  School  recommended  that  the  MAB  aid  another  research  product, 
the  Armor  Combat  Decisions  Game  (CDG) ,  be  developed  by  TRADOC.  Regulations 
were  published  governing  the  use  of  these  training  devices.  NTEC  was  to 
have  action  on  obtaining  the  devices,  in  simplified  form,  per  recommendations 
of  HumRRO.  NTEC  tried,  however,  to  improve  the  tank  model  further  but  en¬ 
countered  problems  with  miniaturization  of  electronic  components.  A  con¬ 
tractor  could  not  be  found  to  build  the  devices  to  specifications;  a  later 
attempt  by  the  Training  and  Doctrine  Command,  Deputy  Chief  of  Staff  for 
Training  to  get  the  devices  built  through  Naval  Training  Device  Center 
(NTDC)  failed  as  well. 

At  present  the  CDG  is  in  use  by  the  Canadian  Army  in  their  own  version, 
successfully.  The  MAB  has  not  yet  been  produced  as  a  training  device.  An¬ 
other  product,  a  map  board  which  was  part  of  project  RECON  for  Armor  train¬ 
ing,  was  given  to  NTDC,  formerly  NTEC,  for  development.  It  is  currently 
confined  in  use  to  the  Ohio  National  Guard. 


4.  Army  Case  Study — ASVAB 


In  February  1966,  the  Assistant  Secretary  of  Defense  for  Manpower  and 
Reserve  Affairs  requested  research  on  a  common  aptitude  battery  that  could 
be  used  by  all  the  services  in  the  high  school  testing  program.  The  Army 
was  designated  lead  service  to  determine  to  what  extent  the  aptitude  tests 
of  the  several  services  were  interchangeable  and  to  develop  an  appropriate 
test  battery.  The  Armed  Services  Vocational  Aptitude  Battery  (ASVAB) ,  con¬ 
sisting  of  a  common  core  of  abbreviated  forms  of  tests  found  to  be  inter¬ 
changeable,  was  a  first  product  of  this  endeavor  and  was  put  into  use  to 
test  potential  recruits  in  the  last  year  of  high  school. 


As  the  original  research  called  only  for  the  development  of  tests  for 
the  high  school  program,  only  those  tests  common  across  all  services  were 
considered.  Thus,  the  service  with  the  smallest  set  drove  the  system. 

From  School  Year  1968/1969  ASVAB  Form  1  was  used  in  high  school  testing; 
in  1972  Forms  2  and  3  were  developed  for  nigh  school  testing,  and  they  also 
became  the  operational  batteries  of  the  Air  Force  and  Marine  Corps.  Army 
bowed  out  as  Executive  Agent  for  ASVAB  research  and  Air  Force  took  over. 

In  the  middle  of  1974,  the  ASD  (M&RA)  decided  that  as  of  1  January  1976 
there  would  be  a  single  classification  battery,  ASVAB,  to  serve  the  primary 


selection  and  classification  purposes  of  all  the  services,  as  well  as  for 
high  school  testing.  The  new  requirement  represented  an  important  change 
of  concept.  To  produce  one  selection  and  classification  battery  to  serve 
needs  of  all  services,  the  service  with  the  largest  set  of  requirements 
drove  the  system,  and  thus,  a  13-test  battery  was  necessary. 

The  battery  was  fielded  1  January  1976,  but  with  the  short  lead  time 
available,  it  was  done  with  no  validation,  a  fact  which  some  Army  observers 
believe  contributed  heavily  to  the  large  attrition  rates  in  TRADOC  schools. 
In  addition,  norms  have  had  to  be  adjusted  and  are  still  being  questioned 
in  the  Army,  Navy,  and  Marine  Corps. 

This  case  study  depicts  the  development  of  a  product  for  which  sponsor 
interest,  enthusiasm,  and  impatience  are  factors  that  have  to  be  dealt  with 
to  delay  (rather  than  hasten)  utilization,  so  that  the  product  will  have  a 
reasonable  chance  to  be  effective  in  operation. 
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APPENDIX  B 


DEFINITIONS  OF  USER,  RESEARCHER,  AND  TRAINING  PRODUCT 


User  Definition 

The  user  is  defined  in  two  dimensions:  organizational  sector  (civilian 
and  military) ,  and  the  functional  level  of  involvement  in  the  Army  training 
technology  transfer  process  (scientific/informative  review,  policy,  adminis¬ 
tration,  implementation,  and  receiver).  These  dimensions  are  depicted  in 
the  User  Identification  Matrix,  which  follows.  Note  that  the  matrix  is  not 
all  inclusive.  It  serves  as  a  starting  point  to  help  determine  what  strategy 
is  needed  to  facilitate  training  technology  transfer  based  on  the  user's 
level  of  involvement  and  sector  in  the  process.  For  example,  attitudec  and 
factors  related  to  the  training  product,  special  group  processes,  and  the 
organizational  structure  are  important  variables  influencing  the  acceptance 
of  an  innovation  (e.g.,  Havelock,  1976;  HIRI,  1976).  Knowing  where  a  par¬ 
ticular  user  is  located  in  terms  of  sector  and  level  of  involvement  will 
provide  information  on  the  user's  perspective  on  attitudes  and  factors  re¬ 
lated  to  the  aforementioned  variables.  If  one  were  interested  in  congres¬ 
sional  subcommittee  support ,  then  the  user ' s  perspective  may  be  couched  in 
terms  of  cost-effectiveness  of  the  training  product,  political  liaison,  and 
organizational  contact  via  a  congressional  staffer.  Or  a  sponsor's  repre¬ 
sentative  may  be  interested  in  the  ability  of  the  training  product  to 
satisfy  the  sponsor's  specific  requirements,  would  be  in  contact  with  the 
researcher  on  an  information  and  interactive  basis  wide  the  researcher,  and 
would  function  as  an  organizational  link  to  promote  change  agentry  with  the 
organization  to  which  the  training  product  is  directed.  It  is  realized 
that  there  may  be  some  degre<  of  overlap  when  categorizing  users  into  levels 
and  sectors,  thus  the  user  rate  within  this  matrix  represents  an  emphasis 
other  than  an  inclusive /exclusive  classification. 

Sources  Key  Words 

Adoption  agency 
Change  agent* 
Client 
Customer 
Linkage  agent* 
Receiver 
Sponsor 


*1  have  narrowed  the  definition  of  these  terms  to  xhe  user  only  (see  Have¬ 
lock  for  expanded  definition) . 


Drucker,  1977 
Havelock,  1976 
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Researcher  Definition 


The  researcher  is  defined  in  two  dimensions:  organizational  sector 
(civilian  and  military) ,  and  the  functional  level  of  involvement  in  the 
RDT&E  of  the  training  product  (application  and  characteristics) .  These 
dimensions  are  depicted  in  a  Researcher  Identification  Matrix,  which  fol¬ 
lows.  This  matrix  represents  the  primary  functions  of  the  researcher (s) 
in  response  to  user  requirements.  An  application  function  would  involve 
translation  of  user  mode  into  a  researchable  question  and  the  monitoring 
of  the  RDT&E  activity  to  insure  that  a  solution  is  found.  Characteristics 
functions  underlie  the  actual  RDT&E  effort  to  produce  the  training  product. 


Sources 


Key  Words 


Drucker,  1977 
Havelock,  1976 


Contract/grant  monitor 
Contractor/grantee 
Developing  agency 
In-house  research  director 
Program/project  director 
Researcher/bench  scientist 
R&D  manager 


Researcher  Identification  Matrix 


Levels 

Sector 

Application3 

Characteristics13 

Civilian 

Contractor 

Military 

Developing  agency 
ARI 

ARI 

Monitoring  and  evaluation  of  RDT&E  activity  to  insure  fulfillment  of  user 
requirements . 

Conduct  "hands-on"  research  activity  to  find  solution  to  researchable 
question. 


Training  Product  Definition 


There  is  a  strong  agreement  ( r_  =  .82)  between  researchers  and  users  in 
what  they  consider  to  be  a  research  produce,  although  a  greater  percentage 
of  researchers  (72%)  check  more  items  qualifying  as  research  products  than 
do  users  (54%)  (Drucker,  1977).  Of  those  items  considered,  an  average  of 
77%  of  both  researchers  and  users  classify  training  products  as  research 
products.  While  a  large  percentage  of  both  researchers  (71%)  and  users 
(74%)  consider  training  programs  and  devices  as  examples  of  training  prod¬ 
ucts,  there  is  disagreement  with  respect  to  classifying  training  literature 
(e.g.,  manuals,  circulars,  instructions).  That  is,  85%  of  the  researchers, 
and  only  53%  of  the  users,  consider  training  literature  as  a  research  prod¬ 
uct  (Drucker,  1977).  Based  on  these  data,  Drucker  (1977)  states  that 
"...  the  user  does  not  give  enough  credit  to  research  in  helping  him  with 
his  implementation  of  policy  changes  or  may  not  accept  that  the  researcher 
makes  a  contribution  with  such  products  as  handbooks  and  manuals."  It  is 
suggested  that  in  order  for  a  training  product  to  be  used  effectively,  the 
user  must  be  shown  (convinced)  how  the  product  can  be  integrated,  used,  and/ 
or  maintained  within  one's  operational  setting.  Training  programs  and  de¬ 
vices  possess  considerable  "face-value"  structure  which  users  can  appreciate 
initially  in  fulfilling  their  training  needs.  However,  training  literature, 
as  a  "stand-alone"  product,  lacks  sufficient  integration  to  solve  training 
needs  from  the  users'  point  of  view.  Optimally,  a  training  product  is  de¬ 
veloped  to  effect  a  change.  Differences  in  viewpoints  as  to  what  specific 
products  accomplish  this  objective  can  be  lessened  if  the  researcher  and 
user  form  a  continuous,  critical  dialog  during  the  training  technology 
transfer  process. 
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Level 


_ Discrepancy _ 

User  Current  Required 

sector  Identification  condition  condition 


Policy 

Civilian 

Congress 

DOD  Secretary 

Military 

TRADOC 

Administration 

Civilian 

DOD  Army 

Military 

FORSCOM/Staff 

Implementation 

Civilian 

X  X  X  X  X 

Military 

Trainer 

Receiver 

Civilian 

X  X  X  X  X 

Military 

Trainee 

aThis  table  provides  a  guide  for  documenting  what  is  needed  if  the  user  at 
each  le\ el  of  involvement  in  the  training  technology  transfer  process  (see 
Kaufman,  1972).  The  table,  when  filled  in,  displays  in  graphic  format  the 
differences  needed  to  be  overcome  for  successful  RDT&E  of  the  training  prod¬ 
uct.  Strategies  could  then  be  developed  to  lessen  these  differences  by 
focusing  on  specific  levels  of  users. 
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APPENDIX  E 


TAEG  REPORT  NO.  40  (DUFFY,  MILLER,  &  STALEY,  1970(b)) 


Section  I 
Introduction 


Background 


The  benefits,  costs,  and  risks  associated  with  introducing  technical  in¬ 
novations  in  education  and  training  can  invoke  large  commitments  of  resources . 
The  rational  assessment  of  payoffs  and  penalties  for  investing  in  changes 
therefore  deserves  technical  attention  with  the  objective  of  arriving  at  sound 
decisions  to  accept  or  reject.  In  essence,  this  means  making  an  analysis  of 
the  full  potential  range  of  applicability  of  the  proposed  innovation  in  terms 
of  benefits,  liabilities,  and  risks,  including  financial  analysis  of  costs, 
and  synthesizing  the  mass  of  resulting  data  into  a  decision-making  presenta¬ 
tion.  Key  judgmental  operations  should,  of  course,  be  retained  by  humans. 
Explicitness  in  procedure  and  in  the  expression  of  human  judgment  is  a  key 
factor  in  rationality.  The  Educational  Technology  Assessment  Model  (ETAM)  is 
a  set  of  comprehensive  procedures  and  variables  for  this  analysis,  synthesis, 
and  decisionmaking.  Although  the  content  of  this  structure,  or  "model,"  is 
directed  specifically  toward  education  and  training,  the  structure  itself  is 
applicable  to  any  rational,  decision-making  context. 

An  innovation  is  broadly  defined.  In  ETAM  it  is  "a  relatively  constant 
or  enduring  change  in  the  procedures,  objects  or  functions  used  in  any  aspect 
of  the  instructional  process  which  may  be  viewed  as  a  benefit  (or  a  liability) 
and  has  associated  costs."  Thus,  an  innovation  may  be  a  technical  invention 
or  it  may  be  a  structural  change  in  the  setting  of  instruction,  such  as  from 
shore-based  to  ship-based  training.  In  summary,  innovations  range  across 
content  of  instruction,  instructional  procedure,  student  selection,  and  gen¬ 
eration  and  implementation  of  training  requirements.* 

The  initial  ETAM  study  developed  a  complete  set  of  manual  procedures, 
parameters,  and  formats  for  all  analysis,  synthesis,  and  decision.  A  com¬ 
prehensive,  descriptive  taxonomy  of  educational  technology  was  generated. 

Its  purpose  was  to  enable  any  proposed  innovation  to  be  described  in  a 
standardized  terminology  for  determining  the  full  range  of  potential  effect 
in  the  Navy:  students,  courses,  jobs,  instructional  devices,  instructional 
development,  as  areas  of  relevance.  An  equivalent  effort  was  spent  in  de¬ 
veloping  or  adapting  analytic  cost  models  applicable  to  the  Navy's  training 
environments. 


*For  additional  reference  to  educational  innovations  see:  Miller,  Robert  B. , 
and  Duffy,  Larry  R. ,  1975.  Design  of  Training  Systems  Phase  II-A  Final  Re¬ 
port  .  TAEG  Report  No.  12-3,  Training  Analysis  and  Evaluation  Group,  ^rlando, 
Fla.  Chapter  III. 


87 


PRECEDING  PAGE  BLAJNK-NOT  FILttED 


The  procedural  model  also  included  return  on  investment  analysis,  sensi¬ 
tivity  analysis,  and  scaling  procedures  for  translating  subjective  evaluations 
of  relative  worth  into  utility  values.  These  utility  expressions  became  in¬ 
puts  to  formal  decision  tree  models  to  be  presented  eo  the  executive  decision¬ 
maker.  A  logical  flow  of  information  is  maintained  from  one  procedural  step 
to  the  next,  so  that  the  assessor  car  readily  review  the  sources  of  benefits 
data,  cost  data,  judgments,  and  assumptions  leading  up  to  the  final  presenta¬ 
tion  of  evaluation  decision  alternatives.  The  decisionmaker  can  change  the 
input  values  to  the  sensitive  parameters  in  the  model  and  determine  the  ef¬ 
fect  on  the  decision  alternatives.  The  decisionmaker  is  not  restricted  to 
working  only  from  the  formal  outcomes  of  staff  evaluations. 


Assumptions  About  Using  the  Model 

Some  assumptions  that  were  stated  in  the  initial  ETAM  study  should  be 
repeated  here. 

1.  The  primary  user  of  ETAM  will  be  the  assessor  of  the  innovation. 

He  (or  the  assessing  team)  have  expertise  in  the  subject  matter 
of  the  particular  innovation  to  be  evaluated.  He  has  mastered 
the  ETAM  classification  structure,  at  least  to  the  level  of  be¬ 
ing  able  to  reference  its  content.  Furthermore,  he  will  have  had 
at  least  several  dozen  hours  of  preliminary  practice  in  applying 
the  procedures  manually  (except  for  calculational  problems)  and 
can  "walk  his  way  through"  the  major  ETAM  tasks.  He  will  also 

have  background  in  the  operational  aspects  of  Navy  training  courses, 
instructional  devices  and  media.  Navy  jobs,  and/or  the  developmental 
stages  of  training;  at  least  he  will  be  familiar  with  those  aspects 
of  these  operations  relevant  to  the  innovation.  This  assumption 
recognizes  that  humans  will  provide  the  information  inputs  and 
judgments,  whereas  the  model  merely  structures,  guides,  and  within 
defined  limits  processes  them. 

2.  The  secondary  user  of  ETAM  will  be  the  executive  decisionmaker. 

He  makes  the  decision  to  commit,  deny,  or  commute  the  resource  for 
implementing  the  decision.  He  may  question  the  constituent  or 
summary  judgments,  evaluations,  data  sources,  and  predictions  of 
the  assessor  embodied  in  the  final  evaluative  recommendation.  He 
can  "peel  back"  the  various  layers  of  data  and  judgments  entering 
into  the  final  calculations.  ETAM  documentation  should  facilitate 
•this  normal  relationship  between  the  executive  and  his  advisory 
staff  work.  The  service  of  the  computer  should  aid  rather  than 
hinder  this  inspection. 

A  key  factor  is  the  ability  to  identify  the  factors  most  sensitive 
to  the  decision  outcome,  and  test  the  range  of  this  sensitivity 
across  the  limit  where  a  recommended  decision  choice  A  changes  to 
a  decision  choice  B. 

3.  The  assessor  will  make  the  final  judgments  as  to  whether  the  inno¬ 
vation  is  or  is  not  applicable  to  an  entity  such  as  a  given  training 
course,  a  given  instruction  vehicle,  a  given  job  or  job-task.  The 
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range-of-effect  search  operations,  including  the  indexing  of  the  in¬ 
novation  and  indexing  the  data  base  content,  and  automatic  searches 
on  data  bases,  will  facilitate  those  judgments,  but  not  replace  them. 

4.  The  user  is  not  compelled  to  apply  the  procedures  beyond  any  stage 
where  common  sense  shows  that  the  outcome  will  be  hopelessly  nega¬ 
tive.  The  model  is  segmented  into  stages  so  as  to  help  this  kind 
of  efficient  termination  come  about.  It  is  easy  to  become  disen¬ 
gaged  from  the  model  with  a  sensibly  completed  piece  of  work.  The 
assessor  is  also  encouraged  to  examine  the  outcome  of  each  stage  of 
work  for  being  within  the  bounds  of  reasonableness. 


Objectives  for  ETAM 

The  design  phase  of  ETAM  led  to  the  specification  of  a  set  of  manual  pro¬ 
cedures  for  assessing  a  proposed  innovation  or  change  in  Navy  training.  A 
proper  follow-on  objective  was  adapting  computerized  aids  for  reducing  the 
large  burdens  of  manual  activities  and  supplementing  rather  than  replacing  or 
interfering  with  human  judgmental  processes.  The  scaling  operations  leading 
to  expressions  of  utility  in  decision  models  were  so  important  that  they  de¬ 
served  intensive  study  of  theory  and  practice  in  behavioral  utility  models 
before  adopting  and  standardizing  on  any  given  procedure.  Another  key  issue 
was  the  practicability  of  the  ETAM  classification  structure  to  the  indexing 
and  searching  of  the  content  in  Navy  data  bases  dealing  with  training  courses, 
jobs,  and  job-tasks,  instructional  vehicles,  and  media.  These  issues  were 
the  basis  for  the  next  phase  of  ETAM. 

The  following  tasks  summarize  the  initial  ETAM  objectives: 

1.  A  study  to  determine  appropriate  scaling  techniques  which  would  in¬ 
crease  the  expected  reliability  and  validity  of  subjective  estimates 
required  within  the  ETAM  procedures.* 

2.  A  study  to  define  indexing  methods  to  provide  equivalence  between 
the  ETAM  range-of-effect  taxonomic  elements  and  data  base  descrip¬ 
tors  for  the  purpose  of  achieving  effective  data  search  and  re¬ 
trieval  operations.** 

3.  A  major  report  presenting  a  comprehensive  overview  of  the  innova¬ 
tions  in  concepts,  methods,  and  practices  that  have  shaped  and  are 
currently  influencing  modern  instructional  technology.  The  purpose 
was  to  give  the  intelligent  layman  an  overview  of  the  effects  of 
innovation  upon  training.  Such  a  document  would  aid  high-ranking 
military  officers  and  business  executives  in  making  decisions  on 
applying  proposed  innovative  techniques  and/or  technologies  to 

*Miller,  Robert  B.,  and  Duffy,  Larry  R. ,  1975.  Design  of  Training  Systems, 

The  Development  of  Scaling  Techniques.  TAEG  Report  No.  32,  Training  Analysis 
and  Evaluation  Group,  Orlando,  Fla. 

**Refer  to  Appendix  A  of  this  report,  TAEG  Report  No.  40. 
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their  training  system.  There  are  parallels,  but  few  direct  equiva¬ 
lents,  of  the  content  of  the  initial  ETAM  report  (TAEG  Report  No. 
12-3) .  It  would  be  an  appropriate  orienting  background  to  the  ETAM 
report  for  those  using  ins  content. 

4.  Computerization  of  the  ETAM  logic  so  that  an  assessor  can  interac¬ 
tively  arrive  at  an  accept/reject  conclusion  using  various  utility, 
probability,  cost,  and  benefit  data  as  model  inputs.  Standard 
program  documentation  and  a  user's  guide  will  be  produced  as  an 
output  of  this  task. 


Section  II 

ETAM  Functional  Requirements 

This  section  of  the  report  has  two  components.  The  first  deals  with  the 
procedures  arising  from  the  steps  in  the  operational  sequence  of  ETAM.  Each 
of  tnese  steps  has  a  set  of  defined  information  inputs,  processing  activities 
and  defined  information  outputs.  This  procedural  definition  was  presented  in 
the  initial  ETAM  report,  but  subject  to  modification  by  the  results  of  the 
studies  on  scaling  and  on  indexing  procedures.  The  second  component  of  the 
functional  requirement  is  the  design  requirement  for  computer  support  to  the 
procedures . 


Procedural  Requirements 


The  ETAM  procedures  consist  of  eight  major  tasks.  The  following  descrip 
tions  will  focus  on  the  data  management  components  of  these  tasks  rather  than 
on  how  they  are  performed  since  these  aspects  are  most  relevant  to  the  func¬ 
tions  of  computerized  data  storage,  processing,  and  retrieval. 

Figure  II-l  is  a  schematic  of  these  steps. 

Task  1 — Formalize  the  Description  of  the  Innovation.  A  project  file  is 
initiated  to  serve  as  a  data  base  for  Innovation  X.  The  innovator  identifies 
the  objectives  of  the  innovation  as  he  conceived  them,  target  applications, 
and  the  results  of  empirical  studies,  if  any,  made  from  the  innovation,  or 
cited  as  relevant  to  the  innovation.  With  the  assistance  of  a  staff  "assess¬ 
or"  with  ETAM  background,  the  prose  description  of  the  innovation  is  indexed 
according  to  the  taxonomic  classification  descriptors  in  ETAM.  The  indexed 
innovation  as  a  set  of  descriptors  will  be  used  as  a  search  specification 
against  data  bases  in  Task  5 — determination  of  range-of-effect.  The  infor¬ 
mation  in  prose  form  will  be  retained  in  a  Task  1  file. 

Task  2 — Develop/Examine  Alternatives  to  the  Innovation.  In  this  task 
the  innovator  (or  other  expert)  is  requested  to  consider  possible  alterna¬ 
tives  to  the  proposed  innovation  which  may  require  a  lesser  level  of  invest¬ 
ment  funding,  and  possibly  be  more  cost  effective.  Any  outcome  of  Task  2 
will  be  treated  procedurally  like  Task  1  and  the  outcomes  of  Task  1.  The 
ultimate  result  will  be  to  create  a  decision  that  compares  Innovation  X  with 
the  alternative  Innovation  XX.  The  profile  of  descriptors  that  indexes 


Innovation  XX  may,  or  may  not,  be  identical  to  the  index  of  Innovation  X. 
Innovation  XX  will  generate  its  own  file  with  a  direct  associative  link 
to  Innovation  X. 

Task  3 — Make  Preliminary  Feasibilicy  Profile.  The  formal  assessment 
procedure  begins  at  this  point  with  a  questionnaire  about  potential  risks 
in  the  implementation,  acceptance,  and  application  of  the  innovation.  The 
issues  deal  with  organizational  incompatibilities,  goal/policies  incompati¬ 
bilities,  technical  support  requirements,  funding  constraints,  and  problems 
in  attitudinal  acceptance  by  users.  A  format  is  available  for  entering 
risk  estimates  and  comments  on  each  of  the  key  risk  variables. 

If  the  risks  are  high,  risk  reduction  projects  are  formulated.  Their 
costs  are  roughly  estimated  and  the  consequent  risk  reduction  is  also  esti¬ 
mated.  If  the  overall  risks  still  seem  excessively  high,  the  decision  to 
reject  the  innovation  from  further  consideration  may  be  made  at  this  point. 

A  format  for  collecting  these  data  is  contained  in  Appendix  B  of  this 
report  (Figure  B-l) .  The  content  becomes  part  of  the  Project  File.  It  will 
be  used  in  Task  4  and  appear  as  essential  data  in  Task  8. 

Task  4 — Perform  Analytic  Feasibility  Assessment.  The  results  of  Task  3 
are  analyzed  in  greater  depth  and  risk  reduction  studies  and  projects  are 
further  defined  and  costed.  A  preliminary  decision  tree  is  structured  for 
providing  initial  guidance  as  to  whether  the  innovation  should  be  accepted 
outright,  accepted  with  the  additional  expenditures  for  the  risk  reduction 
projects,  or  rejected.  Presumably,  a  definitive  range-of-effect  study  of 
potential  benefits  has  not  yet  been  justified,  or  a  sample  of  already  known 
target  applications  for  the  innovation  is  a  sufficient  working  basis  for 
this  stage  of  assessment. 

Project  descriptions  are  prepared  for  each  R&D  effort  with  supporting 
data  about  resource  requirements,  cost  analysis,  and  time  schedules.  These 
projects  are  grouped  into  packages,  each  of  which  is  intended  to  reduce  the 
overall  risk  to  a  reasonable  level.  A  format  for  collecting  cost/saving 
data  is  shown  in  Appendix  B  (Figure  B-2) . 

Decision  trees  are  developed  from  estimated  benefits  data,  cost  data  as¬ 
sociated  with  various  supplemental  projects,  and  risk  estimations.  Refer  to 
Appendix  B  (Figures  B-3  and  B-4) . 

Note  that  at  this  stage,  the  range-of-effect  and  cost  analysis  has  been 
only  grossly  estimated  rather  than  derived  from  a  full  scale  analysis.  But 
even  on  these  bases,  the  differences  among  the  decision  alternatives  may  be 
so  large  and,  based  on  sensitivity  analysis,  appear  so  reliable,  that  a  de¬ 
cision  may  be  justified  without  further  analysis. 

If  analysis  proceeds  further,  the  risk  estimations  and  risk  reduction 
project  data  are  fed  into  Task  6. 

Task  5 — Determine  Range-of-Ef feet.  The  prior  tasks  have  been  concerned 
with  the  assessment  of  the  innovation  over  a  limited  range  of  application. 
Initially,  it  was  the  target  applications  identified  by  the  innovator;  in 
Task  4  a  preliminary  extended  range-of-application  was  considered.  Task  5 
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enables  the  assessor  to  apply  both  the  formal  descriptors  applied  to  index  mg 
the  innovation  and  the  contextual  knowledge  about  the  innovation  to  the  full 
range  of  the  Navy's  inventory  of  training  courses,  instruction  vehicles,  and 
jo'--tasks . 

ETAM  stipulates  that  entities — training  courses,  job-tasks,  instructional 
vehicles — subject  to  range-of-effect  analysis  may  be  indexed  according  to  the 
rules  of  the  descriptor  taxonomy  in  ETAM  formulated  in  Appendix  A  of  this  re¬ 
port.  These  indexed  entities  comprise  a  data  base.  This  data  base  can  be 
searched  by  search  arguments  composed  of  the  descriptors  that  uniquely  identify 
the  relevant  properties  of  the  innovation . 

Thus,  the  indexed  description  of  the  innovation  that  was  made  in  Task  1 
becomes  an  input  into  range-of-effect  search.  When  the  search  arguments  reveal 
hits  in  che  data  base,  the  assessor  examines  contextual  information  about  the 
entity  (a  course,  a  job-task,  or  an  instructional  vehicle)  and  makes  a  judgment 
of  relevance  or  irrelevance.  When  matches  between  the  innovation's  properties 
and  the  entity's  properties  are  made,  the  assessor  estimates  the  kind  and  pro¬ 
portional  magnitude  of  benefit/liability  that  is  likely  to  be  contributed  by 
the  innovation.  A  method  from  the  initial  ETAM  study  for  describing  a  benefit 
(or  liability)  is  shown  in  Appendix  B  (Figure  B-5) .  Ordinarily,  there  will  be 
a  number  of  benefit  variables  and  liability  variables  that  make  up  a  pattern 
or  profile  applicable  to  the  innivation's  promise.  The  identification  of  af¬ 
fected  entities  (courses,  etc.)  is  input  to  cost-benefits  analysis,  Task  6. 

Task  6 — Perform  Cost-Benefits  Analysis.  The  decision  tree  developed  in 
Task  4  is  refined  to  include  more  precise  costs  and  savings  derived  from  pro¬ 
cessing  the  tangible  benefits  through  the  model.  Thus,  if  the  innovation  has 
been  estimated  to  enable  an  average  reduction  of  20%  for  learning  the  content 
of  Course  A  to  criterion,  the  cost  model  determines  how  frequently  the  course 
is  taught,  how  many  students  take  the  course  and,  from  its  base  of  cost  data 
about  Course  A,  computes  in  dollars  the  actual  projected  savings.  In  addi¬ 
tion,  the  assessor  uses  utility  scaling  techniques  for  analyzing  intangible 
benefits  so  that  they  are  expressed  in  "equivalent  dollars,"  thus  enabling 
them  to  be  combined  into  a  single  continuum  of  worth  or  value.  Equivalent 
dollars  is  a  utility  expression  rather  than  a  literal  dollar  value.  Proba¬ 
bilities  of  implementation  success  and  user  acceptance  with  and  without  the 
risk  reduction  projects  are  refined  and  the  decision  variables  are 
recalculated. 

The  procedural  model  described  in  the  ETAM  Phase  II-B  report,  "Design  of 
Training  Systems,  The  Development  of  Scaling  Procedures,"  is  the  structure 
whereby  the  assessor  generates  multivariate  utilities  for  outcomes  in  the 
decision  tree. 

The  model  permits  sensitivity  analysis  of  variables  that  could  reasonably 
change  enough  to  affect  the  choice  of  a  decision  alternative. 

The  output  of  this  Task  goes  into  Task  7 ,  and  subsequently  to  the  de¬ 
cisionmaker  in  Task  8. 

Task  7 — Perform  Financial  Analysis.  This  task  is  concerned  with  assessing 
the  tangible  benefits  and  liabilities  (those  expressible  in  real  dollars)  in 
terms  of  certain  economic  measures.  The  investment  costs  and  the  annual  costs 
and  savings  are  calculated  over  a  planning  period  extending  a  number  of  years 
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into  the  future.  Rates-of-return  on  the  invested  dollars  are  determined  for 
the  incremental  effect  of  each  alternative  compared  to  the  primary  project. 
Alternatives  consist  of  the  proposed  innovation,  the  existing  system,  and  any 
other  approaches  defined  in  Task  2  which  were  considered  reasonable  candi¬ 
dates  for  further  assessment.  This  assessment  process  provides  a  separate, 
distinct  view  of  the  value  of  the  innovation  from  that  gained  from  the  de¬ 
cision  tree,  assessment  in  Task  6.  Both  are  inputs  to  the  decisionmaker. 

The  model  enables  sensitivity  analysis.  The  purpose  is  to  give  the  as¬ 
sessor  insight  into  the  variables  that  could  cause  a  change  in  the  decision 
if  they  were  to  vary  over  a  reasonably  expected  range. 

Task  8 — Make  the  Accept/Reject/Study  Decision.  The  immediate  bases  for 
the  making  of  the  decision  are  the  financial  analysis,  plus  sensitivity 
analysis,  from  Task  7,  and  the  decision  tree  data,  plus  sensitivity  analysis, 
from  Task  6. 

However,  the  organized  content  of  the  data  base  files  of  the  assessment 
project  enables  the  decisionmaker  to  examine  any  of  the  constituent  elements 
beneath  the  summary  presentation  made  to  him.  He  may  substitute  his  own 
evaluations  of  worth,  probability  of  outccmes,  importance  of  intangibles, 
estimates  of  benefits  or  liabilities.  He  nay  "peel  back"  the  data  in  each 
of  the  seven  tasks  by  selectively  accessing  the  files  on  each  of  these 
tasks.  He  could  examine  samples  of  range-of-effect  entities  contained  in 
computer  files  and  retrieved  interactively. 

The  executive  is  thus  in  a  position  to  put  probes  behind  the  facade  of 
conclusions  presented  to  him.  He  is  therefore  capable  of  reassurance  in  the 
results  or  direct  participation  in  changing  them  according  to  his  own  values 
and  store  of  information. 

Comment .  The  preceding  description  is  merely  a  synoptic  outline  of  the 
ETAM  procedure.  It  is  neither  a  substitute  nor  replacement  for  the  full  de¬ 
scription,  including  stipulations,  assumptions,  and  caveats,  that  are  con¬ 
tained  in  the  source,  the  initial  ETAM  report,  TAEG  Report  No.  12-3. 
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APPENDIX  V 


WELSH ’ S  DISSEMINATION  METHODOLOGY 


Chapter  III 

Rationale  for  the  Purpose  and  Steps  of  the  Methodology 

3.1  Introduction 

The  purpose  of  this  chapter  is  to  provide  a  detailed  justification  for 
the  purpose  and  each  of  the  steps  in  the  methodology.  Literature  from  such 
sources  as  rural,  medical,  and  general  sociology;  economics  and  marketing; 
and  education  will  be  cited  in  support  of  the  steps  included. 

Actually,  the  bulk  of  research  reported  in  knowledge  diffusion  and  uti¬ 
lization  is  concentrated  in  sociclogy  and  anthropology.  Some  work  hat  been 
done  in  marketing  research,  although  this  is  a  much  newer  field.  Relatively 
little  dissemination  research  has  been  completed  in  education.  The  method¬ 
ology  represents  an  attempt  to  synthesize  the  best  available  knowledge  from 
the  various  disciplines  mentioned. 

In  the  following  pages,  the  purpose,  and  all  the  steps  of  Draft  I  of  the 
dissemination  methodology  will  be  listed.  This  will  be  followed  in  turn  by 
the  justification  of  the  purpose  and  the  rationale  for  including  each  of  the 
ten  major  steps. 

3.2  Dissemination  Methodology:  Draft  I 

Purpose:  To  meet  needs  through  the  dissemination  of  products. 

Case  I:  The  disseminator  is  working  for  a  product  developer  (a  special 
case — the  disseminator  is  the  product  developer) 

Case  II:  The  disseminator  is  working  as  an  independent  change  agent 

(i.e.,  his/her  remuneration  would  come  from  something  like  a 
university  salary;  dissemination  is  not  his/her  only  major 
concern;  rather,  one  of  a  number  of  interests) 

Case  III:  The  disseminator  is  working  for  a  funded  agency  whose  function 
is  to  disseminate  products  (for  example,  the  Far  West  Labora¬ 
tory  for  Educational  Research  &  Development) 

Case  IV:  The  disseminator  is  working  for  a  consumer  or  group  of  con¬ 
sumers  (e.g.,  a  school  system) 


preceding  page  BLANK-NOT  FILMED 
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I :  Negotiate  a  contract  with  a  product  developer  interested  in 
dissemination 


A.  Explain  each  major  step  in  the  methodology  to  the  product 
developer 

B.  Identify  the  product  to  be  disseminated 

C.  Identify  the  resources  available  for  the  dissemination  effort 

D.  Prepare  the  contract  and  secure  the  product  developer's  final 
approval 

II:  Plan  the  implementation  of  the  remaining  steps  in  the  methodology 

III:  Have  the  pioduct  developer  design — or  adapt,  if  the  product  is  al¬ 
ready  designed — the  product  to  be  as  amenable  to  dissemination  as 
possible,  without  changing  the  character  of  the  product 


A.  Determine  the  resources  available  for  this  step 

B.  Make  an  initial  judgment  as  to  what  general  populations  benefit 

from  the  adoption  of  the  product 

C.  Make  the  product  as  compatible  with  the  potential  adopter's 

values,  culture,  and/or  traditions  as  possible 

1.  Determine  the  values,  culture,  and/or  traditions  of  the  po¬ 
tential  adopters 

2.  Determine  the  adaptability  of  the  product 

3.  Adapt  the  product  to  the  values,  culture  and/or  traditions 
of  the  potential  adopters 

D.  Keep  the  cost  of  the  product  as  low  as  possible 

1.  If  product  costs  nothing  or  almost  nothing  (e.g.,  a  research 
report  advocating  some  variety  of  behavior  change) ,  move  to 
Step  III .E. 

2.  Break  the  product  down  into  component  parts  if  possible 

3.  Determine  which  of  the  components  are  essential  to  the  prod¬ 
uct  if  it  is  to  accomplish  the  purpose  for  which  it  was 
designed 

4.  Eliminate  those  components  found  to  be  nonessential  in 
Step  3 

5.  Continue  to  break  down  the  components  until  it  is  relatively 
easy  to  determine  the  lowest  possible  cost  for  each.  The 
total  will  then  be  the  lowest  possible  cost  for  the  product 

6.  Document  cost  information  for  use  in  Step  V 
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Reduce  the  complexity  of  the  product  as  much  as  possible 


1.  Steps  III.D.2.  through  III.D.4.  will  have  yielded  compo¬ 
nents  of  the  product.  If  the  components  are  broken  down 
as  far  as  possible,  go  to  Step  3 

2.  Break  down  the  components  into  their  most  basic  subcomponents 

3.  If  necessary,  provide  explanation  of  the  final  list  of  com¬ 
ponents  of  the  product 

4 .  Document  complexity  information  for  use  in  Step  V 

Make  the  product  "divisible,"  so  that  it  can  be  tried  initially 
on  a  small  scale 

1.  Determine  whether  the  product  is  divisible  or  can  be  made 
divisible  without  sacrificing  its  ability  to  accomplish  its 
purpose.  If  it  is  not,  or  cannot  be  made  divisible,  go  to 
Step  III.G. 

2 .  Determine  how  the  product  can  be  tried  on  a  limited  basis 

a.  Determine  whether  only  part  of  the  product  need  be  tried 

b.  Determine  whether  only  a  part  of  the  adopting  popula¬ 
tion  (given  that  it  is  made  up  of  more  than  one  person) 
needs  to  try  the  product  to  give  it  a  fair  trial 

c.  Document  all  possible  ways  the  product  can  be  made  divis¬ 
ible  for  use  in  Step  V 

Make  the  product  observable,  if  possible,  so  that  a  potential 
adopter  can  see  it  in  operation  before  he  makes  his  decision 

1.  Determine  whether  any  institutions  already  use  the  product 

2.  Determine  whether  the  product  developer  or  the  disseminator 
can  demonstrate  the  product 

3.  Document  observability  for  use  in  Step  V 

Devise  appropriate  support  services  which  the  adopter  may  avail 
himself  of  after  adoption  of  the  product 

1.  Determine  potential  difficulties  adopters  can  encounter  when 
using  the  product 

2.  Determine  which  of  these  can  be  eliminated,  or  at  least  re¬ 
duced,  by  providing  support  services  to  the  adopter 

3.  Plan  specifically  support  services  to  reduce  problems  identi¬ 
fied  in  Step  2 
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IV: 


Identify  general  populations  that  will  benefit  from  the  adoption  of 
the  product  (potential  adopters) 

A.  Determine  the  resources  available  for  this  step 

B.  Identify  general  populations  that  have  a  need  for  the  product 

1.  Determine  all  populations  that  could  possibly  have  a  need 
for  the  product 

a.  Read  the  relevant  literature 

b.  Talk  with  people  whose  work  is  in  related  areas 

c .  Brainstorm  all  possible  general  populations 

2.  Determine  if  the  general  populations  identified  in  Step  IV.B.l 
actually  need  the  product 

a.  Read  relevant  literature  on  these  populations 

b.  Talk  with  experts  on  these  populations 

c.  Sample  opinions  from  the  populations  themselves 

d.  Conduct  relevant  research  on  these  populations 

3.  Compile  a  list  of  populations  that  are  identified  as  needing 
the  product 

C.  Among  these  populations,  identify  those  sub-populations  for  whom 
the  product  fills  a  high-priority  need 

1.  Implement  the  needs  analysis  methodology,  using  at  least  a 
sample  of  the  target  sub-population 

2.  Determine  whether  or  not  the  need  the  product  fills  has  a 
sufficiently  high  priority  on  the  needs  of  the  population; 
if  it  does,  go  to  Step  IV. D.;  if  not,  select  another  sub¬ 
population  and  implement  needs  analysis  again 

D.  Of  these,  identify,  as  far  as  possible,  those  sub-populations 
on  whom  the  product  would  have  seriously  detrimental  side  ef¬ 
fects,  and  leave  them  out  of  the  dissemination  effort 

1.  If  the  resources  are  relatively  small,  make  judgment  from 
existing  relevant  knowledge 

a.  Brainstorm  possible  side  effects 

b.  Talk  to  people  knowledgeable  about  those  sub-populations 

c.  Read  relevant  literature  on  those  sub-populations 

d.  Sample  opinions  from  the  sub-populations 
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E.  The  above  steps  will  result  in  a  set  of  potential  adopters  who 
will  be  the  target  population;  if  it  is  different  from  the 
group  identified  in  Step  III.B.,  consider  whether  or  not  you 
need  to  recycle  from  Step  III.C.  on 

V.  Identify,  among  the  designated  potential  adopters,  those  subgroups 
most  likely  to  react  favorably  to  the  product  and  focus  communica¬ 
tion  upon  them 


A.  Determine  the  resources  available  for  this  step 

B.  Determine  those  in  the  population  who  are  the  early  adopters 

1.  Decide  on  definition  of  "early  adopter" 

2.  Identify  products  used  by  the  target  population  similar  to 
the  product  to  be  disseminated 

3.  Determine  those  in  target  population  who  have  a  record  of 
early  adoption  of  those  products 

a.  Examine  available  records  of  adoption  of  those  products 

b.  Talk  with  those  who  use  those  products 

c.  Talk  with  those  connected  with  the  adoption  of  those 
products 

4.  Compile  a  list  of  those  identified  as  "early  adopters" 

C.  If  resources  are  relatively  large,  and  if  there  are  a  relatively 
large  number  of  early  adopters,  determine  the  opinion  leaders 
among  the  early  adopters.  If  not,  go  to  Step  V.D. 

1.  Use  other  sociometric  devices  to  identify  opinion  leaders 
(e.g.,  questionnaires  that  ask,  "name  the  three  colleagues 
from  whom  you  would  be  most  apt  to  seek  advice  with  regard 
to  (whatever  the  nature  of  the  product  is)") 

2.  If  the  disseminator  has  insufficient  expertise  in  interpret¬ 
ing  sociometric  devices  (if  sophisticated  sociometric  devices 
are  in  fact  used) ,  employ  an  appropriate  consultant 

3.  Compile  a  final  list  of  those  members  of  the  target  popula¬ 
tion  to  be  the  first  at  whom  dissemination  efforts  will  be 
directed 

D.  Develop  a  professional  level  (as  opposed  to  friendship  level)  of 
rapport  with  the  potential  adopter  identified  in  Step  V.B.4.  or 
Step  V.C.3. 

1.  Observe  common  rules  of  courtesy  carefully  (punctuality , 
politeness,  etc.) 
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VI: 


2.  Remain  honest  and  as  objective  as  possible  at  all  times 

3.  Be  aware  of  the  potential  adopter's  professional  activities, 
or  the  activities  of  his/her  institution 

4.  Make  your  interest  (if  genuine)  in  his/her  activities  or 
those  of  his/her  institution  known  to  the  potential  adopter 

5.  Explain  clearly  to  the  potential  adopter  that  your  intent  is 
ta  disseminate  the  product  only  to  meet  needs.  If  he/she 
does  not  see  that  it  meets  a  need,  you  are  not  interested 

in  disseminating  the  product  to  him/her 

6.  Explain  fully  your  role  in  disseminating  the  product 

7.  Be  able  to  explain  readily  any  aspect  of  the  product 

E.  Explain  the  product  fully,  and  describe  how  it  will  meet  the  po¬ 
tential  adopter's  needs 

1.  Explain  your  perception  of  the  potential  adopter's  needs  (or 
the  needs  of  his/her  system).  If  the  potential  adopter's 
diagnosis,  and  if  the  potential  adopter  and  the  disseminator 
cannot  reach  an  agreement  on  needs,  go  to  another  potential 
adopter.  Otherwise,  proceed  to  Step  2 

2.  Explain  your  perception  of  what  the  total  impact  of  the  prod 
uct  will  be  on  the  potential  adopter's  system 

a.  Explain  how  you  think  it  will  meet  need(s) 

b.  Explain  what  negative  effects  may  result 

3.  Explain  the  characteristics  of  the  product  that  were  de¬ 
termined/developed  in  Step  IV 

a.  Explain  the  cost  of  the  product 

b.  Explain  how  the  product  can  be  observed  in  use  (if  it 
can) 

c.  Explain  how  the  product  can  be  tried  on  a  limited  basis 
(if  it  can) 

d.  Explain  its  compatibility  with  the  cultures,  values,  and 
traditions  of  the  potential  adopter  (if  it  is,  in  fact, 
compa  tible ) 

e.  Explain  the  support  services  available  for  use  if  the 
product  is  adopted 

If  the  potential  adopter (s)  decide (s)  to  adopt,  make  the  product 
available  to  him/her  as  soon  as  possible,  including  all  available 
support  services  if  they  are  desired 
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VII:  If  resources  for  this  step  remain,  implement  the  "2-step  model" 
i.e.,  help  the  opinion  leaders  disseminate  the  product  to  others 
in  the  population 

A.  Determine  whether  the  opinion  leader  wants  to  help  in  the  dis¬ 
semination  effort 

B.  Determine  whether  the  opinion  leader  is  to  be  trusted  with  the 
resources  available  for  this  step.  If  not,  go  to  Step  VIII 

C.  Determine  how  much  and  what  kinds  of  resources  the  opinion 
leader  needs 

D.  Make  the  resources  available  to  the  opinion  leader 
VIII:  Evaluate  the  results  of  the  adoption/rejection 

A.  The  Fortune-Hutchinson  evaluation  methodology  is  recommended 
with  the  product  developer  as  the  decisionmaker 

B.  If  adopted,  evaluate  its  acceptance,  use,  and  impact,  including 
unintended  outcomes 

1.  If  it  meets  the  adopter's  need,  proceed  with  other  potential 
adopters  in  the  same  manner — i.e.,  return  to  Step  IV 

2.  If  it  does  not  meet  the  need,  or  for  some  other  reason  causes 
trouble  for  the  adopter,  return  to  Step  III 

C.  If  rejected,  evaluate  reason(s)  for  rejection  and  return  to 
Step  III  or  IV,  as  the  product  developer  decides  (i.e.,  he  may 
choose  either  to  redesign  his  product  or  to  aim  the  existing 
product  at  a  different  target  population) 

IX:  Proceed  through  Steps  IV-VIII  until  the  product  is  completely  dissemi¬ 
nated,  or  until  resources  run  out 

X:  Evaluate  the  success  of  the  methodology  and  revise  where  appropriate 
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APPENDIX  G 


SUPPLEMENTAL  INFORMATION  NEEDED  FOR  EVALUATION 


To  broaden  the  scope  of  the  evaluation  of  technology  transfer  in  the  Army., 
supplemental  information  should  be  obtained.  The  types  of  information  needed 
are  suggested  below: 

1.  Within  ARI ,  one  could  evaluate  research  products  in  general  (of 
which  training  products  form  a  subset) .  An  initial  step  here  is  to  categor¬ 
ize  those  products  with  which  ARI  is  involved.  For  example,  one  could  sub¬ 
jectively  "factor  analyze"  the  research  products  reported  by  Drucker  (1977) 
into  four  major  areas,  based  on  their  purpose  (see  Table  G.l) .  Then,  within 
each  category,  one  could  focus  interest  on  completed  products  and  those  m 
the  process  of  being  completed  (i.e.,  handed  off  from  6.3  to  6.4  budgetary 
and  operational  control).  Within  each  of  the  completed  products,  determine 
successful  (S)  versus  nonsuccessful  (N)  products.  Success  can  be  determined 
by  objective  sources  (e.g.,  training  effectiveness  data,  usage  in  terms  of 
frequency  counts)  and  by  subjective  sources  (e.g.,  obtaining  statements  from 
different  levels  of  users) .  Within  the  process  products,  select  products 
that  have  a  very  high  probability  of  being  successful  (+)  and  those  that 
may  or  may  not  be  successful  (?) .  Finally,  track  (rate)  all  the  products 

in  terms  of  the  organizational  events  based  on  Yin  et  al.  (1978)  approach 
(e.g.,  incorporation  and  maintenance  of  products  over  time).  The  completed 
products'  outcomes  will  be  predictive  markers  for  the  process  products'  out¬ 
comes.  That  is,  if  the  completed  S  and  N  products  cluster  discriminate ly  on 
Yin's  organizational  events,  then  this  institutionalization  approach  may  be 
used  to  predict  the  outcome  of  the  process  +  and  ?  products.  The  relevance 
of  this  approach  is  that  it  may  help  to  highlight  the  processes  that  occur 
during  the  handoff  of  ARI  products  to  the  user  from  6.3  to  6.4  RDT&E  funding 
and  operational  control.  If  the  "6. 3-6. 4"  interval  is  critical  for  technology 
transfer,  then  perhaps  application  of  Yin's  "routinization"  approach  may  help 
to  correlate  organizational  events  with  S  and  N  products.  This  proposed  analy¬ 
sis  is  schematized  in  Table  G.2. 

It  is  realized  that  this  proposed  analysis  takes  ARI  out  of  its  formal 
RDT&E  boundary  (past  6.3)  ,  but  if  ARI  focuses  only  on  the  "6.3  handoff"  (user 
acceptance)  while  still  being  held  accountable  for  its  institutionalization 
(6. 4-6. 7) ,  events  influencing  this  accountability  (i.e.,  effecting  the  out¬ 
come  of  the  S  and  N  of  products)  may  be  unnoticed  during  the  6. 3-6. 4  interval. 

A  major  implication  of  previous  assignment  of  accountability  to  the  HR  RDT&E 
community  (see  Note  2)  is  that  the  handling  of  research  products  through  the 
6.4  to  6.7  phases  does  not  cause  significant  transformations  on  the  eventual 
outcomes  associated  with  the  use  of  the  product.  Therefore,  serious  consid¬ 
eration  should  be  given  to  studying  those  events  that  not  only  correlate  with 
later  incorporation  (6.4  to  6.7),  but  also  those  during  the  6. 3-6. 4  interval 
(i.e.,  the  utilization  stage  of  the  institutionalization  activity). 

2.  Supplemental  to  the  first  suggestion  would  be  information  depicting 
those  Army/DOD  agencies  that  use  (e.g.,  spend)  RDT&E  funds,  and  the  types  of 
RDT&E  funds  used.  This  suggestions  is  a  matrix  consisting  of  6.1  through  6.7 
funds  listed  along  one  dimension,  and  the  names  of  the  agencies  along  the 
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Table  G.l 


Proposed  Categories  of  ARI  Research  Products 


1.  Informative-Explanatory  ("What  is") 

Data  banks 

Job  aid,  information 
Making  the  researcher  an  expert 
New  technologic  al  information 
Project  report 

Report  suitable  for  publication  in  professional  journal  (Scientific 
Report) 

Research  findings  (data) 

Research  material  for  university-level  education 

2.  Informative-Procedural  ("How  to") 

Guidelines 
Handbook 
Policy  manual 

SOP  (Standing  operating  procedures) 

Technical  manual 

Training  literature  (manuals,  circulars,  instructions) 

3.  Recommendations  (this  is  a  subset  of  both  1  and  2  above,  depending  on  its 

purpose) 

Recommendations  for  change  in  doctrine 
Recommendations  for  change  in  policy 

4-  Research  Methodology 

Research  tool 

Tests  and  measurement  irstruments 

5 .  Training  Products 

Training  literature  (also  listed  under  2) 

Training  device 
Training  program 


Table  G.2 


Tracking  the  Outcome  of  Selected  Research  Products 


Research  products 


History 


Outcome 


Completed 


Success 

Nonsuccess 


Informative-explanatory 


Process 


_+ 


Completed 


£ 

N 


Informative-procedural 

4* 

Process  - — 


Research  methodology 


Completed 


Process 


+ 

y 


Training  product 


Completed 


_S 

N 


Process 


_+ 

9 


Note:  Document  (rate)  products  on  the  organizational  events  during  institu¬ 
tionalization  as  listed  by  Yin  et  al.  (1978) . 


105 


other  dimension.  One  could  look  at  agencies  involved  in  HR  RDT&E  research 
products  in  general,  or  one  could  delimit  the  field  of  interest  to  training 
products.  The  question  of  interest  is  from-whom-to-whom  are  funds  handed 
off  during  the  transition  of  RDT&E  funding  control  of  the  research  product. 
This  information  would  indicate  how  much  interaction  one  should  have  with 
specific  agencies  who  are  involved  in  the  handoff  of  budgetary  and  operational 
control  of  the  research  product.  For  example,  one  could  see  how  many  agencies 
spend/use  funds  during  the  6. 3-6. 4  interval.  This  matrix  would  serve  as  a 
“funding  flow"  chart  consisting  of  clusters  of  agencies  which  may  be  corre¬ 
lated  with  organisational  events  critical  to  the  outcome  of  the  research 
product . 


3.  An  a  priori  baseline  of  success  rates  of  research  product  utilization 
would  help  to  guide  subsequent  funding  and  research  directions.  Determining 
the  baseline  rate  has  not  been  addressed  empirically.  First,  there  is  a  defi¬ 
nition  issue.  This  could  be  resolved  by  using  both  objective  and  subjective 
sources  of  information,  as  mentioned  previously.  Then  the  scope  must  be  ad¬ 
dressed.  The  scope  could  be  limited  up  to  6.3  (user  acceptance),  or  could  in¬ 
clude  the  transition  period  (6. 3-6. 4  interval)  of  budgetary  and  operational 
control.  Also,  a  within-Army  standard  could  be  referenced  from  the  "hardware" 
RDT&E  community  (e.g.,  the  Harry  Diamond  Labs),  using  their  "success"  rates 
for  comparison  with  those  of  "software"  RDT&E  products. 

It  is  suggested  that  the  above-mentioned  supplemental  information,  as 
well  as  the  proposed  evaluation  scheme,  may  be  incorporated  with  the  product 
utilization  activity  of  the  Plans,  Programs,  and  Operations  (PPO)  office  of 
ARI.  The  above-mentioned  suggestions  might  provide  a  conceptual  framework 
from  which  the  PPO  office  could  decide  future  evaluative  efforts  on  research 
product  utilization. 


106 


APPENDIX  H 


FORMAT  FOR  AN  IMPLEMENTATION  PLAN* 


Requirements  Analysis 

1.  General  Nature  of  the  Problem 

a.  Description  of  current  training  problem 

b.  What  operational  problems  exist 

2.  Reasons  for  Attention  at  This  Time 

a.  Congressional  concern 

b.  Budgetary  reasons 

c.  Military  readiness 

d.  As  appropriate 

3.  Objectives  Related  to  the  Problem 

a.  Use  established  statement  of  objectives,  or,  where  necessary,  a  ver¬ 
sion  of  it  specially  tailored  to  the  operational  problem. 

b.  Three  essential  components 

(1)  Good  effect  or  result  intended  to  be  achieved 

(2)  For  whom 

(3)  By  what  means 

4.  Measure  of  Effectiveness  (MOE's)  Related  to  the  Objective 

a.  If  the  plan  relates  to  a  particular  training  program,  use  the  estab¬ 
lished  MOE's  supplemented  as  necessary  by  others  more  directly  related 
to  the  hypothesis  being  tested. 

b.  Attempt  to  use  quantitative  indicators  of  objectives  results. 

c.  Don't  confuse  MOE's  with  activity  levels. 

5.  Target  Groups  -  Receiver  Level  of  User 

a.  Present  level  of  training  effectiveness 

b.  Estimated  potential  level  of  training  effectiveness 

6.  Beneficiary  Group  -  Any /All  Level (s)  of  User 


*Extracted  and  modified  from  Keller  (1979) . 


7.  Programs  Related  to  or  Affected  by  this  Problem 

a.  Local  agency 

b .  Command 

c.  As  appropriate 

8.  Alternative  Considered 


Dissemination  and  Institutionalization 


9.  How  the  Approved  Alternative  Will  Operate  (the  detailed  operational 
plan) 

a.  Staffing 

b.  Organization 

c.  Relations  with  other  organizations 

d.  Policies 

e.  Method  of  operating 

(1)  Chronological  account  of  a  typical  operation 

(2)  Special  techniques  used 

10.  Estimated  Effectiveness  of  the  Approved  Alternative 

a.  Refer  back  to  the  MOE's  in  4,  above. 

b.  Refer  back  to  the  discussion  in  the  General  Nature  of  the  Problem 
in  1,  above. 

11.  Estimated  Costs  of  the  Approved  Alternative 


a.  Investment 

■= 

(1)  R&D 

(2)  Facilities 

(3)  Equipment 

(4)  Initial  training 

b.  Operating 

(1)  Personnel 

(2)  Minor  equipment 

(3)  Supplies 

(4)  Direct  services 

(5)  Contracted  services 


As  applicable 


c.  Totals — Annual 
Totals — 10  year 

(1)  Indicate  whether  the  costs  are  in  constant  or  current  dollars. 

(2)  Indicate  whether  present  value  discounting  has  been  done. 
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12.  Spillover  Effects 


a.  Refer  to  the  public  agencies  listed  in  7,  above. 

b.  Other  spillovers 

c.  Coordination  problems  between  levels  of  users 

13.  Other  Considerations 

a.  Resistance 

b.  Constraints  on  operations 

c.  Linkage  agent  role  for  sponsor’s  representative 

d.  Training 

e.  Continued  fiscal  backing 


RDT&E  Concerns 

14.  Proposed  Method  of  Evaluating  the  Approved  Alternative 

a.  In  drawing  up  an  evaluation  scheme  (or  research  design) ,  one  must 
allow  for  certain  basic  concerns,  to  make  judgments  subsequently 
about  the  overall  validity  of  the  evaluation.  These  include 

(1)  The  nature  of  the  questions  to  be  answered  or  the  decisions  to 
to  be  made. 

(2)  What  will  constitute  a  "success"  or  a  "failure." 

(3)  The  basic  methodology  to  be  employed,  including  consideration 
of  the  bases  of  comparison  and  the  adequacy  and  power  of  the 
specific  techniques  to  be  used. 

(4)  The  level  of  confidence  which  can  be  attached  to  the  conclusions 

(5)  The  generalizability  of  the  conclusions. 

b.  In  dealing  with  these  basic  concerns  the  following  more  specific 
procedures  and  questions  should  be  dealt  with  in  full  detail: 

(1)  What  are  the  hypotheses  to  be  tested? 

(a)  How  do  they  relate  to  the  stated  objectives? 

(b)  What  are  the  logical  linkages  between  the  overall  good  ef¬ 
fect  and  the  immediately  testable  good  effects? 

(c)  Are  there  assumptions  which  can  or  should  be  converted  to 
testable  hypotheses? 

(2)  What  measures  of  effectiveness  are  related  to  the  hypotheses  to 
be.  tested? 

(a)  Are  these  direct  measures  of  effectiveness  or  are  they 
proxy  measures? 


(b)  If  they  are  proxy  measures,  what  is  the  basis  in  evidence 
for  the  presumed  relationship  between  the  proxy  measure 
and  the  direct  measure? 

(3)  Data  on  what  other  measurable  consequences  and  items  of  evalua¬ 
tive  interest  need  to  be  collected? 

(4)  How  frequently  are  data  on  the  measures  of  effectiveness,  other 
measurable  consequences,  and  other  items  of  evaluative  interest 
to  be  gathered? 

(5)  For  how  long  will  each  category  of  data  be  collected? 

(6)  How,  exactly,  will  all  of  these  data  be  collected? 

(a)  By  whom;  and  will  they  i.eed  any  special  training  in  data 
collection  procedures?  What  opportunities  for  bias  are 
there  among  the  data  collectors? 

(b)  What  procedures  will  be  followed  in  collecting  the  data? 

Do  they  create  the  possibility  of  bias  creeping  in? 

(c)  What  forms  will  be  used  to  collect  the  data?  Are  they 
clear  enough?  Will  Privacy  Act  requirements  inhibit  the 
data  collection  effort? 

(7)  With  respect  to  the  experimental  group: 

(a)  What  criteria  will  be  used  to  select  subjects  into  the 
experimental  group? 

(b)  How  will  the  sample  be  stratified? 

(c)  What  will  be  the  maximum  and  minimum  usable  sample  sizes? 

(d)  What  methods  or  mechanisms  will  be  used  for  actually  se¬ 
lecting  subjects  in  accordance  with  the  criteria  specified 
in  a,  b,  and  c,  above? 

(e)  To  what  degree  can  the  Hawthorne  Effect  be  eliminated? 

(8)  With  respect  to  the  control  group  used  as  a  basis  of  comparison 
with  the  experimental  group: 

(a)  What  exactly  will  be  the  form  of  comparison? 

(b)  How  will  points  7a,  b,  c,  and  d,  above,  be  dealt  with  for 
uhe  control  group? 

(c)  Are  there  ethical  or  legal  problems  involved  in  choosing  or 
using  the  control  group? 
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(9)  Exactly  what  calculations  will  be  made  using  the  data  to  be 
gathered? 

(a)  Are  the  statistical  or  other  techniques  to  be  used  appro¬ 
priate  to  the  problem? 

(b)  Are  they  sufficiently  powerful  to  produce  the  desired 
results? 

(c)  Have  all  of  the  data  required  to  drive  the  computational 
models  been  identified? 

(d)  Have  provisions  been  made  to  assure  the  availability  of 
all  of  these  data? 

(e)  If  some  of  the  data  cannot  be  collected  at  all,  or  with 
sufficient  reliability,  what  fall-back  computational  pro¬ 
cedures  can  be  used? 

(10)  Meeting  what  criteria,  or  ranges  of  values  which  result  from 

the  calculations,  will  be  considered  as  constituting  a  "success" 
an  "indeterminate  result"?  a  "failure"? 
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